Methods and Apparatus for Supporting Content Generation, Transmission and/or Playback

ABSTRACT

Methods and apparatus for supporting the capture of images of surfaces of an environment visible from a default viewing position and capturing images of surfaces not visible from the default viewing position, e.g., occluded surfaces, are described. Occluded and non-occluded image portions are packed into one or more frames and communicated to a playback device for use as textures which can be applied to a model of the environment where the images were captured. An environmental model includes a model of surfaces which are occluded from view from a default viewing position but which maybe viewed is the user shifts the user&#39;s viewing location. Occluded image content can be incorporated directly into a frame that also includes non-occluded image data or sent in frames of a separate, e.g., auxiliary content stream that is multiplexed with the main content stream which communicates image data corresponding to non-occluded environmental portions.

RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/917,333 filed Jul. 30, 2020, which is a continuation of U.S.patent application Ser. No. 16/040,325 filed Jul. 19, 2018, which is acontinuation of U.S. patent application Ser. No. 15/058,125 filed Mar.1, 2016, which claims the benefit of U.S. Provisional Application Ser.No. 62/126,701 filed Mar. 1, 2015, U.S. Provisional Application Ser. No.62/126,709 filed Mar. 1, 2015, and U.S. Provisional Application Ser. No.62/127,215 filed Mar. 2, 2015, each of which is hereby expresslyincorporated by reference in its entirety.

FIELD

The present invention relates to methods and apparatus for generating,transmitting and/or using image content which maybe communicated in oneor more different content stream formats.

BACKGROUND

Content capture methods normally focus on the capture of content from asingle location in an environment. Various surfaces maybe obstructed,e.g., occluded from view, from the single image capture location. Forexample the back of a column may not be visible for the inside of a boxmay not be visible.

As environmental simulations become more realistic, users of displaydevices would like to be able to move in the simulated environment. Forexample if they stand up they may expect to be able to look over the topof a box and see the inside of the box which was not visible from adefault viewing location in the environment corresponding to theposition from which images were captured in the environment. Similarlyif a user changes his head location by leaning to the side the usermight expect to be able to peer around a column.

Unfortunately, if images captured from the single location in theenvironment is the only content provided to the playback device theplayback device will be unable to show the previously occluded imagecontent which a user expects to see when the user changes his locationin the environment that is being simulated using the images capturedfrom the single location in the environment. Thus, there is a need forcapturing, communicating and/or using images of occluded portions of anenvironment to facilitate more realistic environmental simulations.While there is a need for improvements with regard to transmission aswell as playback, all features and embodiments need not address both thetransmit side and the playback side and features which provideimprovements to either would be useful and beneficial.

From the above it should be appreciated that there is a need for methodsand/or apparatus which would allow images of occluded portions of anenvironment to be captured and at least some portions of the images ofthe occluded portions of the environment to be provided to a playbackdevice.

While not necessary or critical for all embodiments, it would be usefulif one or more methods of transmitting occluded image content along withnon-occluded image content to a playback device could be supported in arelatively data efficient manner.

From the above it should also be appreciated that there is a need formethods and/or apparatus which would allow a playback device to receiveand/or use images of non-occluded portions of an environment along withat least some image content corresponding to occluded portions of theenvironment.

SUMMARY

Methods and apparatus for processing captured images and transmittingthe images in one or more formats to support playback are described. Theimage processing and transmission in some embodiments is performed in amanner that facilitates playback and allows for a playback device toprovide a user a simulation of being present in the environment wherethe images were captured. In some but not necessarily all embodimentssufficient content is transmitted to a playback device to allow a userto change his/her position and for the displayed content to reflect, atleast partially, the change in the viewers position so that the user cansee content that was not visible from a default position in theenvironment.

Thus, at least some features are directed to methods and apparatus forsupporting a range of viewing positions in a simulated environment usinga playback device are described. In various embodiments an environmentis simulated from a first, e.g., default, viewing location. The usershead position, e.g., head orientation and location, is monitored andchanges from the default viewing location, e.g., due to a shift in theuser's head location, are detected. A user may choose to move, e.g.,shift, his head to a new viewing position which is within a range ofsupported viewing locations relative to the location to which thedefault viewing position corresponds. Turns of the head left or rightand up/down are supported by changing which portions of the environmentare displayed to the user with the displayed portions corresponding tothe default viewing position but taking into consideration differenthead angles. While the user's head remains at the default viewingposition, image content captured by one or more stereoscopic camerapairs positioned at the default viewing position within the actualenvironment are rendered and displayed to the user. Image rendering inthe context of a default viewing position may, and sometimes doesinvolve displaying images using a 3D model of the environment, a firstcontent stream including frames, e.g., pairs of left and right eyeimages, corresponding to the default viewing position and a UV map usedfor rendering frames corresponding to images from the default viewingposition content stream onto the 3D model.

While detected head rotations will result in the display of differentportions of the content stream corresponding to the default viewingposition, a user may alter his/her head position by leaning, e.g., leftor right, forward or back and/or standing up or sitting down. Suchchanges represent a physical shift or offset from the default viewingposition, not simply a head rotation, and result in a change in viewinglocation.

In the case of a physical offset from a default viewing position, a usermay expect to see a portion of the environment which was occluded fromview from the default viewing position. For example, if there was asupport column positioned to the left of the user's default viewinglocation, by leaning forward and thus by changing the user's viewingposition, e.g., location and/or direction of view, relative to thesupport column the user may expect to be able to see a portion of theenvironment which was previously occluded from view. The portion of theenvironment that was occluded from view will normally not be availablefrom the cameras, e.g., one or more pairs of cameras used to captureleft and right eye views corresponding to the default viewing positionsince the column will normally have blocked the capture of such imageareas to the cameras used to capture the images corresponding to thedefault viewing position.

In accordance with various embodiments, additional cameras, beyond thoseused to capture the images used to generate the content stream for thedefault viewing area are used and capture portions of the environmentoccluded from view from the default viewing area. The cameras may bemounted on a camera rig that includes one or more camera pairs used tocapture left and right eye views corresponding to the default viewingposition. In some such embodiments the cameras used to capture theoccluded image areas are of a different type or types than the camerasof the stereoscopic camera pair(s). For example the cameras used tocapture the occluded image areas maybe light field cameras or lowerresolution cameras included in the camera rig at different positionsthan the camera pairs used to capture left and right eye positionsimages for stereoscopic playback. In some embodiments the imagescaptured by the light field camera or cameras is used to provideenvironmental depth information allowing for real time mapping of theenvironment and detection of changes in the environment during an eventwhile also providing images that can be used to supplement the imagescaptured by the stereoscopic camera pairs in the event a user changeshis/her head position from the default viewing position.

While in some embodiments the image data corresponding to environmentalportions occluded from view from the default viewing position arecaptured by one or more cameras located at various locations on the samecamera rig on which the camera pairs used to capture the stereoscopicimages corresponding to the default viewing position, in otherembodiments the image portions are captured by cameras at otherlocations within the environment of interest or the images are capturedat times different from the time at which the images provided ascorresponding to the default viewing position are captured. For example,the images corresponding to the occluded image portions may be capturedby one or more cameras located to the left or right of the camera rigused to capture images corresponding to the default viewing position.Alternatively, if it known that a temporary impediment to viewing fromthe default viewing position during an event, am image may be capturedprior to placement of the temporary impediment, e.g., banner orequipment storage rack added for the event, and then used as image datato support viewing of the occlude image area. While in various someembodiments the image data corresponding to the occluded image area maynot correspond to a different time period than the image datacorresponding to the default viewing position, the occluded image areais not likely to convey important scene information and a user in manycases will not realize the image data for the normally occlude imagearea is not from the same point in time as other areas of the displayed3D scene.

In various embodiments occluded image content, e.g., image content notvisible from the default viewing location, is communicated to a playbackdevice in addition to non-occluded image content.

Various methods for communicating the occluded and non-occluded imagecontent to a playback device are supported. An individual embodimentneed not support multiple ways of transmitting and/or receiving occludedand non-occluded image content but in some embodiments the contentserver and playback device support multiple method of communicating andreceiving such information.

In some embodiments occluded and non-occluded image content is packedtogether into frames which are transmitted to a playback device.

In another embodiment occluded image content is communicated in adifferent content stream, e.g., a primary content stream, which isseparate from an auxiliary content stream which is used to communicateoccluded image content. The primary and auxiliary content streams maybeand often are multiplexed into a program stream used to communicatecontent corresponding to a program, e.g., sporting event, concert orother event that may occur in an environment such as a stadium, concerthall, etc. which can be modeled.

In order to display images corresponding to normally occluded imageareas, in some embodiments a supplemental image content stream isprovided in some but not all embodiments to a playback device to provideimage content corresponding to image areas occluded from view from thedefault viewing position. Such supplemental content may be in the formof a frame including content corresponding to one, but in many cases, aplurality of normally occluded areas of the environment. In addition tothe images corresponding to the occluded image portions, an occludedimage portion UV map is provided in at least some embodiments withinformation indicating how segments of a frame providing occluded imagedata are to be mapped to the 3D model of the environment. Which segmentsof an occluded image portion from are used may, and in some embodimentsdo, depend on the change to the default viewing position made by a user.Changes in the default viewing position up to a predetermined amount maybe supported. For example moving the viewing position up to a foot ormore left or right may be supported though the use of the occluded imageportions provided by the supplemental image data. The image datacorresponding to occluded image portions can be, and in some embodimentsis, sent in a frame at the same or a lower frame rate than image datacorresponding to the default viewing position. In some embodimentschanges in captured images corresponding to normally occluded portionsof the environment are detected and a new supplemental frame is sent inresponse to the detected change to provide updated occluded area imagecontent to be displayed by the playback device if needed. Occluded imageportions may and sometimes do have the same resolution of images of theenvironment corresponding to the default viewing position.

However, in other embodiments the images corresponding to normallyoccluded portions of the environment may be of lower resolution thanthose captured by the camera pairs used to capture images correspondingto the default viewing position. This is often the case when imagescaptured by one or more cameras which use a light field array, such asLytro cameras, are used to capture images of normally occluded imageportions.

While an image processing and/or transmission system may support one ormore methods of communicating non-occluded image content and/or occludedimage content, all transmission systems need not support all the methodsdescribed herein.

A playback device in various embodiments is capable of receiving and/orusing non-occluded image content and occluded image content. Theplayback device may receive such content from the system describedherein which serves such content.

Various features are directed to a playback device and/or a method ofoperation a playback device.

In one embodiment a playback device receives frames including bothoccluded and non-occluded image portions. In another embodiment aplayback device receives frames, e.g., primary frames, including imagedata corresponding to the default viewing position and frames, e.g.,auxiliary frames, providing image data corresponding to normallyoccluded portions of the environment were are not viewable from thedefault viewing position in the environment. The two different ways inwhich occluded content can be received correspond to different contentstream formats.

In response to detecting a user change from the default viewing positionto a new viewing position the playback device generates and display animage including a first image portion generated from received contentcorresponding to the default viewing position and at least a secondimage portion generated from received image content corresponding to anormally occluded image portion.

The content corresponding to the normally occluded image portion isreceived in some embodiments in a frame which includes both non-occludedand occluded image portions. In other embodiments occluded image data isreceived in an auxiliary which provides images of environmental areasnormally occluded from view from the default viewing position. Theselection of which portion or portions of the normally occluded imageswhich is displayed is determined based on a detected offset of the userfrom the default viewing position.

By supping occluded and non-occluded image content multiple viewingpositions, including some corresponding to different locations in theenvironment, can be supported during content playback.

While numerous features and embodiments have been described in thesummary it should be appreciated that not all embodiments require orinvolve use of all of the above described features and that someembodiments may include one or a few of the above described featuresand/or support one of the above described methods of communicationand/or using occluded image portions. Numerous additional features,embodiments and benefits are discussed in the detailed description whichfollows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a camera rig implemented in accordance with oneembodiment along with a calibration target which may be used to forcalibrating the camera rig.

FIG. 2 illustrates the camera rig with three pairs of cameras, e.g., 3pairs of cameras capturing stereoscopic image data, mounted in thecamera rig.

FIG. 3 illustrates an exemplary camera rig with an exemplary protectivecover implemented in accordance with some exemplary embodiments.

FIG. 4 illustrates another exemplary camera rig implemented inaccordance with some exemplary embodiments.

FIG. 5 illustrates yet another exemplary camera rig implemented inaccordance with some exemplary embodiments.

FIG. 6 illustrates an exemplary system implemented in accordance withsome embodiments of the invention.

FIG. 7A is a first part of FIG. 7 which illustrates a flowchart of anexemplary method of operating an exemplary processing system inaccordance with some embodiments.

FIG. 7B is a second part of FIG. 7 which illustrates a flowchart of anexemplary method of operating the processing system.

FIG. 7C is a third part of FIG. 7 which illustrates a flowchart of anexemplary method of operating the processing system.

FIG. 7D is a fourth part of FIG. 7 which illustrates a flowchart of anexemplary method of operating the processing system.

FIG. 7 comprises the combination of FIGS. 7A, 7B, 7C and 7D.

FIG. 8 illustrates the steps of an exemplary content serving routinewhich is implemented in some embodiments as part of the method offlowchart of FIG. 7.

FIG. 9 illustrates a first stream format which is used to serve, e.g.,stream or download content which does not include occluded imageportions.

FIG. 10 illustrates a second stream format which is used to serve, e.g.,stream or download content, which includes non-occluded image portionsand occluded image portions in a frame.

FIG. 11 illustrates a third stream format which is used to serve, e.g.,stream or download content, which includes non-occluded image portionsbeing transmitted in frames of a main or primary content stream andoccluded image portions in an auxiliary stream.

FIG. 12 illustrates an exemplary processing system implemented inaccordance with an exemplary embodiment.

FIG. 13A is a first part of FIG. 13 which illustrates a flowchart of anexemplary method of operating an exemplary rendering and playback devicein accordance with an exemplary embodiment.

FIG. 13B is a second part of FIG. 13 which illustrates a flowchart of anexemplary method of operating the rendering and playback device.

FIG. 13 comprises the combination of FIGS. 13A and 13B.

FIG. 14 illustrates the steps of an exemplary first stream formatplayback routine which is implemented by the playback device of thepresent invention as part of performing the method of FIG. 13.

FIG. 15A is a first part of FIG. 15 which illustrates the steps of anexemplary second stream format playback routine which is implemented bythe playback device as part of performing the method of FIG. 13.

FIG. 15B is a second part of FIG. 15 which illustrates the steps of theexemplary second stream format playback routine implemented by theplayback device as part of performing the method of FIG. 13.

FIG. 15 comprises the combination of FIGS. 15A and 15B.

FIG. 16A is a first part of FIG. 16 which illustrates the steps of anexemplary third stream format playback routine which is implemented bythe playback device as part of performing the method of FIG. 13.

FIG. 16B is a second part of FIG. 16 which illustrates the steps of theexemplary third stream format playback routine implemented by theplayback device as part of performing the method of FIG. 13.

FIG. 16C is a third part of FIG. 16 which illustrates the steps of theexemplary third stream format playback routine implemented by theplayback device as part of performing the method of FIG. 13.

FIG. 16D is a fourth part of FIG. 16 which illustrates the steps of theexemplary third stream format playback routine implemented by theplayback device as part of performing the method of FIG. 13.

FIG. 16 comprises the combination of FIGS. 16A, 16B, 16C and 16D.

FIG. 17 illustrates an exemplary 3D environmental mesh model that may beused in various embodiments with a plurality of nodes illustrated as thepoint of intersection of lines used to divide the 3D model intosegments.

FIG. 18 illustrates an exemplary UV map that can be used for mappingportions of a 2D frame, providing a texture, to the mesh model of FIG.17.

FIG. 19 illustrates an exemplary rendering and playback deviceimplemented in accordance with an exemplary embodiment.

DETAILED DESCRIPTION

Various features relate to the field of panoramic stereoscopic imageryand various imaging devices and/or apparatus, e.g., camera rigsincluding cameras, discussed below are well suited for capturinghigh-definition, high dynamic range, high frame rate stereoscopic,360-degree panoramic video using a minimal number of cameras in anapparatus of small size and at reasonable cost while satisfying weight,and power requirements for a wide range of applications.

Stereoscopic, 360-degree panoramic video content is increasingly indemand for use in virtual reality displays. In order to producestereoscopic, 360-degree panoramic video content with 4K or greater ofresolution, which is important for final image clarity, high dynamicrange, which is important for recording low-light content, and highframe rates, which are important for recording detail in fast movingcontent (such as sports), an array of professional grade, large-sensor,cinematic cameras or other cameras of suitable quality are often needed.

In order for the camera array to be useful for capturing 360-degree,stereoscopic content for viewing in a stereoscopic virtual realitydisplay, the camera array should acquire the content such that theresults approximate what the viewer would have seen if his head wereco-located with the camera. Specifically, the pairs of stereoscopiccameras should be configured such that their inter-axial separation iswithin an acceptable delta from the accepted human-model average of 63mm. Additionally, the distance from the panoramic array's center pointto the entrance pupil of a camera lens (aka nodal offset) should beconfigured such that it is within an acceptable delta from the acceptedhuman-model average of 101 mm.

In order for the camera array to be used to capture events and spectatorsports where it should be compact and non-obtrusive, it should beconstructed with a relatively small physical footprint allowing it to bedeployed in a wide variety of locations and shipped in a reasonablesized container when shipping is required.

The camera array should also be designed such that the minimum imagingdistance of the array to be small, e.g., as small as possible, whichminimizes the “dead zone” where scene elements are not captured becausethey fall outside of the field of view of adjacent cameras.

It would be advantageous if the camera array can be calibrated foroptical alignment by positioning calibration targets where the highestoptical distortion is prone to occur (where lens angles of viewintersect AND the maximum distortion of the lenses occur). To facilitatethe most efficacious calibration target positioning, target locationsshould, and in some embodiments are, determined formulaically from therig design.

FIG. 1 shows an exemplary camera configuration 100 used in someembodiments. The support structure shown in FIGS. 4 and 5 is not shownin FIG. 1 to allow for better appreciation of the camera pairarrangement shown used in some embodiments. While in some embodimentsthree camera pairs are used such as in the FIG. 1 example in some butnot all embodiments a camera array, e.g., the camera positions of therig, is populated with only 2 of the 6-total cameras which may be usedto support simultaneous 360-degree stereoscopic video. When the camerarig or assembly is configured with less than all 6 cameras which can bemounted in the rig, the rig is still capable of capturing thehigh-value, foreground 180-degree scene elements in real-time whilemanually capturing static images of the lower-value, background180-degree scene elements, e.g., by rotating the rig when the foregroundimages are not being captured. For example, in some embodiments when a2-camera array is used to capture a football game with the field of playat the 0-degree position relative to the cameras, the array is manuallyrotated around the nodal point into the 120-degree and 240-degreepositions. This allows the action on the field of a sports game ormatch, e.g., foreground, to be captured in real time and the sidelinesand bleachers, e.g., background areas, to be captured as stereoscopicstatic images to be used to generate a hybridized panorama includingreal time stereo video for the front portion and static images for theleft and right rear portions. In this manner, the rig can be used tocapture a 360 degree view with some portions of the 360 view beingcaptured at different points in time with the camera rig being rotatedaround its nodal axis, e.g., vertical center point between the differentpoints in time when the different view of the 360 scene area arecaptured. Alternatively, single cameras may be mounted in the second andthird camera pair mounting positions and mono (non-stereoscopic) imagecontent captured for those areas.

In other cases where camera cost is not an issue, more than two camerascan be mounted at each position in the rig with the rig holding up to 6cameras as in the FIG. 1 example. In this manner, cost effect cameradeployment can be achieved depending on the performance to be capturedand, the need or ability of the user to transport a large number, e.g.,6 cameras, or the user's ability to transport fewer than 6 cameras,e.g., 2 cameras. In some embodiments an environmental depth map isgenerated from the images captured by the cameras in the camera rig 100.

FIG. 1 depicts a six (6) camera assembly 100 also sometimes referred toas a rig or camera array, along with a calibration target 115. Thecamera rig 100 illustrated in FIG. 1 includes a support structure (shownin FIGS. 4 and 5) which holds the cameras in the indicated positions, 3pairs 102, 104, 106 of stereoscopic cameras (101, 103), (105, 107),(109, 111) for a total of 6 cameras. The support structure includes abase 720 also referred to herein as a mounting plate (see element 720shown in FIG. 4) which supports the cameras and to which plates on whichthe cameras are mounted can be secured. The support structure maybe madeof plastic, metal or a composite material such as graphite orfiberglass, and is represented by the lines forming the triangle whichis also used to show the spacing and relationship between the cameras.The center point at which the doted lines intersect represents thecenter nodal point around which the camera pairs 102, 104, 106 can berotated in some but not necessarily all embodiments. The center nodalpoint corresponds in some embodiments to a steel rod or threaded centermount, e.g., of a tripod base, around which a camera support framerepresented by the triangular lines can be rotated. The support framemay be a plastic housing in which the cameras are mounted or tripodstructure as shown in FIGS. 4 and 5.

In FIG. 1, each pair of cameras 102, 104, 106 corresponds to a differentcamera pair position. The first camera pair 102 corresponds to a 0degree forward to front facing position and normally meant to cover theforeground where the main action occurs. This position normallycorresponds to the main area of interest, e.g., a field upon which asports game is being played, a stage, or some other area where the mainaction/performance is likely to occur. The second camera pair 104corresponds to a 120 degree camera position (approximately 120 degreefrom the front facing) degree position) and is used to capture a rightrear viewing area. The third camera pair 106 corresponds to a 240 degreeviewing position (approximately 240 degree from the front facing) and aleft rear viewing area. Note that the three camera positions are 120degrees apart.

Each camera viewing position includes one camera pair in the FIG. 1embodiment, with each camera pair including a left camera and a rightcamera which are used to capture images. The left camera captures whatare sometimes referred to as a left eye images and the right cameracaptures what is sometime referred to as right eye images. The imagesmay be part of a view sequence or still image captured at one or moretimes. Normally at least the front camera position corresponding tocamera pair 102 will be populated with high quality video cameras. Theother camera positions may be populated with high quality video cameras,lower quality video cameras or a single camera used to capture still ormono images. In some embodiments the second and third camera embodimentsare left unpopulated and the support plate on which the cameras aremounted is rotated allowing the first camera pair 102 to capture imagescorresponding to all three camera positions but at different times. Insome such embodiments left and right rear images are captured and storedand then video of the forward camera position is captured during anevent. The captured images may be encoded and streamed in real time,e.g. while an event is still ongoing, to one or more playback devices.

The first camera pair 102 shown in FIG. 1 includes a left camera 101 anda right camera 103. The left camera has a first lens assembly 120secured to the first camera and the right camera 103 has a second lensassembly secured to the right camera 103. The lens assemblies 120, 120′include lenses which allow for a wide angle field of view to becaptured. In some embodiments each lens assembly 120, 120′ includes afish eye lens. Thus each of the cameras 102, 103 can capture a 180degree field of view or approximately 180 degrees. In some embodimentsless than 180 degrees is captured but there is still at least someoverlap in the images captured from adjacent camera pairs in someembodiments. In the FIG. 1 embodiment a camera pair is located at eachof the first (0 degree), second (120 degree), and third (240 degree)camera mounting positions with each pair capturing at least 120 degreesor more of the environment but in many cases with each camera paircapturing 180 degrees or approximately 180 degrees of the environment.

Second and third camera pairs 104, 106 are the same or similar to thefirst camera pair 102 but located at 120 and 240 degree camera mountingpositions with respect to the front 0 degree position. The second camerapair 104 includes a left camera 105 and left lens assembly 122 and aright camera 107 and right camera lens assembly 122′. The third camerapair 106 includes a left camera 109 and left lens assembly 124 and aright camera 111 and right camera lens assembly 124′.

In FIG. 1, D represents the inter-axial distance of the first 102stereoscopic pair of cameras 101, 103. In the FIG. 1 example D is 117 mmwhich is the same or similar to the distance between pupils of the leftand right eyes of an average human being. Dashed line 150 in FIG. 1depicts the distance from the panoramic array's center point to theentrance pupil of the right camera lens 120′ (aka nodal offset). In oneembodiment corresponding to the FIG. 1 which example the distanceindicated by reference number 150 is 315 mm but other distances arepossible.

In one particular embodiment the footprint of the camera rig 100 isrelatively small. Such a small size allows the camera rig to be placedin an audience, e.g., at a seating position where a fan or attendancemight normally be located or positioned. Thus in some embodiments thecamera rig is placed in an audience area allowing a viewer to have asense of being a member of the audience where such an effect is desired.The footprint in some embodiments corresponds to the size of the base towhich the support structure including, in some embodiments a centersupport rod is mounted or support tower is located. As should beappreciated the camera rigs in some embodiments can rotate around thecenter point of the base which corresponds to the center point betweenthe 3 pairs of cameras. In other embodiments the cameras are fixed anddo not rotate around the center of the camera array.

The camera rig 100 is capable of capturing relatively close as well asdistinct object. In one particular embodiment the minimum imagingdistance of the camera array is 649 mm but other distances are possibleand this distance is in no way critical.

The distance from the center of the camera assembly to the intersectionpoint 151 of the views of the first and third camera parts represents anexemplary calibration distance which can be used for calibrating imagescaptured by the first and second camera pairs. In one particularexemplary embodiment, an optimal calibration distance, where lens anglesof view intersect and the maximum distortion of the lenses occur is 743mm. Note that target 115 may be placed at a known distance from thecamera pairs located at or slightly beyond the area of maximumdistortion. The calibration target include a known fixed calibrationpattern. The calibration target can be and is used for calibrating thesize of images captured by cameras of the camera pairs. Such calibrationis possible since the size and position of the calibration target isknown relative to the cameras capturing the image of the calibrationtarget 115.

FIG. 2 is a diagram 200 of the camera array 100 shown in FIG. 1 ingreater detail. While the camera rig 100 is again shown with 6 cameras,in some embodiment the camera rig 100 is populated with only twocameras, e.g., camera pair 102 including cameras 101 and 103. As shownthere is a 120 degree separation between each of the camera pairmounting positions. Consider for example if the center between eachcamera pair corresponds to the direction of the camera mountingposition. In such a case the first camera mounting position correspondsto 0 degrees, the second camera mounting position corresponds to 120degrees and the third camera mounting position corresponding to 240degrees. Thus each camera mounting position is separated by 120 degrees.This can be seen if the center line extending out through the center ofeach camera pair 102, 104, 106 was extended and the angle between thelines measured.

In the FIG. 2 example, the pair 102, 104, 106 of cameras can, and insome embodiments do, rotate around the center point of the camera rigallowing for different views to be captured at different times withouthaving to alter the position of the camera rig base. That is, thecameras can be rotated around the center support of the rig and allowedto capture different scenes at different times allowing for a 360 degreescene capture using the rig shown in FIG. 2 while it is populated withonly two cameras. Such a configuration is particularly desirable from acost perspective given the cost of stereoscopic cameras and is wellsuited for many applications where it may be desirable to show abackground captured from the same point of view but at a different timethan the time at which the front scene including the main action duringa sporting event or other event may occur. Consider for example thatduring the event objects may be placed behind the camera that it wouldbe preferable not to show during the main event. In such a scenario therear images may be, and sometimes are, captured prior to the main eventand made available along with the real time captured images of the mainevent to provide a 360 degree set of image data.

Various features also relate to the fact that the camera supportstructure and camera configuration can, and in various embodiments does,maintain a nodal offset distance in a range from 75 mm to 350 mm. In oneparticular embodiment, a nodal offset distance of 315 mm is maintained.

The support structure also maintains, in some embodiments an overallarea (aka footprint) in a range from 400 mm² to 700 mm². In oneparticular embodiment, an overall area (aka footprint) of 640 mm² ismaintained. The support structure also maintains a minimal imagingdistance in a range from 400 mm to 700 mm. In one particular embodiment,a minimal imaging distance of 649 mm is maintained. In one particularembodiment the optimal calibration distance of the array is where lensangles of view intersect AND the maximum distortion of the lenses occur.In one particular exemplary embodiment this distance is 743 mm.

As discussed above, in various embodiments the camera array, e.g., rig,is populated with only 2 of the 6-total cameras which would normally berequired for simultaneous 360-degree stereoscopic video for the purposeof capturing the high-value, foreground 180-degree scene elements inreal-time while manually capturing static images of the lower-value,background 180-degree scene elements.

FIG. 3 shows an exemplary camera rig 300 which is the same or similar tothe rig of FIGS. 1 and 2 but without a support tripod and with a plasticcover 350 placed over the camera pairs. The plastic cover 350 includeshandles 310, 312, 314 which can be used to lift or rotate, e.g., whenplaced on a tripod, the camera rig 300. The camera rig 300 is shown withthree pairs of cameras, a first camera pair 302 including cameras 301,303 with lens assemblies 320, 320′, a second camera pair 304 includingcameras with lens assemblies 322, 322′, and a third camera pair 306including cameras with lens assemblies 324, 324′. The plastic cover 350is secured to the mounting platform 316, which may be implemented as aflat plate with one or more slots and screw holes. The plastic cover 350is secured to the base with nuts or screws 330, 331 which can be removedor tightened by hand to allow for easy removal or attachment of thecover 350 and easy access to the cameras of the camera pairs. While sixcameras are included in the rig 300 shown in FIG. 3, a single camerapair may be included and/or a single camera pair with one or moreindividual cameras located at the other camera mounting positions wherethe camera pairs are not mounted may be used.

FIG. 4 illustrates a drawing 800 showing one view of an exemplary camerarig 801 implemented in accordance with some exemplary embodiments. Anarray of cameras is included in the camera rig 801 some of which arestereoscopic cameras. In the illustrated view of the camera rig 801 indrawing 800, only a portion of the camera rig 801 is visible while asimilar arrangement of cameras exist on the other sides (also referredto as different faces) of the camera rig 801 which cannot be fully seenin the drawing 800. In some but not all embodiments, the camera rig 801includes 13 cameras secured by a top plastic body or cover 805 and abottom base cover 842. In some embodiments 8 of these 13 cameras arestereoscopic cameras such as the cameras 804, 806, 812 and 814 in pairswhile many other cameras are light field cameras such as cameras 802 and810 which are visible in the drawing 800 and cameras 815 and 820 whichare not fully but partially visible in drawing 800. Various othercombinations of the cameras are possible. In some embodiments a camera825 is also mounted on the top portion of the camera rig 801, e.g., topface 840 of camera rig 801, to capture images of a top hemisphere of anenvironment of interest. The plastic body/cover 805 includes handles811, 813, 817 which can be used to lift or rotate the camera rig 801.

In some embodiments the camera rig 801 includes one light field camera(e.g., camera 802) and two other cameras (e.g., cameras 804, 806)forming a stereoscopic camera pair on each longer side of the camera rig801. In some such embodiments there are four such longer sides (alsoreferred to as the four side faces 830, 832, 834 and 836) with eachlonger side having one light field camera and one stereoscopic camerapair, e.g., light field camera 802 and stereoscopic camera pair 804, 806on one longer side 836 to the left while another light field camera 810and stereoscopic camera pair 812, 814 on the other longer side 830 tothe right can be seen in drawing 800. While the other two side faces arenot fully shown in drawing 800, they are shown in more detail in FIG. 8.In some embodiments at least some of the cameras, e.g., stereoscopiccameras and the light field cameras, in the camera rig 801 use a fisheye lens. In various embodiments each of the cameras in the camera rig801 is protected by a corresponding lens/camera guard to protect thecamera and/or lens against a physical impact and/or damage that may becaused by an object. For example cameras 802, 804 and 806 are protectedby guards 845, 847 and 849 respectively. Similarly cameras 810, 812 and814 are protected by guards 850, 852 and 854 respectively.

In addition to the stereoscopic camera pair and the light field cameraon each of the four side faces 830, 832, 834 and 836, in someembodiments the camera rig 801 further includes a camera 825 facing inthe upward vertical direction, e.g., towards the sky or another topceiling surface in the case of a closed environment, on the top face 840of the camera rig 801. In some such embodiments the camera 825 on thetop face of the camera rig 801 is a light field camera. While not shownin drawing 800, in some other embodiments the top face 840 of the camerarig 801 also includes, in addition to the camera 825, anotherstereoscopic camera pair for capturing left and right eye images. Whilein normal circumstances the top hemisphere (also referred to as the skyportion) of a 360 degree environment, e.g., stadium, theater, concerthall etc., captured by the camera 825 may not include action and/orremain static in some cases it may be important or desirable to capturethe sky portion at the same rate as other environmental portions arebeing captured by other cameras on the rig 801.

While one exemplary camera array arrangement is shown and discussedabove with regard to camera rig 801, in some other implementationsinstead of just a single light field camera (e.g., such as cameras 802and 810) arranged on top of a pair of stereoscopic cameras (e.g.,cameras 804, 806 and 812, 814) on four faces 830, 832, 834, 836 of thecamera rig 801, the camera rig 801 includes an array of light fieldcameras arranged with stereoscopic camera pair. For example in someembodiments there are 3 light field cameras arranged on top of astereoscopic camera pair on each of the longer sides of the camera rig801. In another embodiment there are 6 light field cameras arranged ontop of stereoscopic camera pair on each of the longer sides of thecamera rig 801, e.g., with two rows of 3 light field cameras arranged ontop of the stereoscopic camera pair. Moreover in another variation acamera rig of the type shown in drawing 800 may also be implemented suchthat instead of four faces 830, 832, 834, 836 with the cameras pointedin the horizontal direction, there are 3 faces of the camera rig withcameras pointing in the horizontal direction.

In some embodiments the camera rig 801 may be mounted on a supportstructure such that it can be rotated around a vertical axis. In variousembodiments the camera rig 801 may be deployed in an environment ofinterest, e.g., such as a stadium, auditorium, or another place where anevent to be captured is taking place. In some embodiments the lightfield cameras of the camera rig 801 are used to capture images of theenvironment of interest, e.g., a 360 degree scene area of interest, andgenerate depth maps which can be used in simulating a 3D environment anddisplaying stereoscopic imaging content.

FIG. 5 illustrates a drawing 1100 showing a view of yet anotherexemplary camera rig 1101 implemented in accordance with some exemplaryembodiments. The exemplary camera rig 1101 is similar to the camera rig801 in most and many aspects and includes the same or similarconfiguration of cameras as discussed with regard to camera rig 801above. The camera rig 1101 includes four side faces 1130, 1132, 1134,1136 and a top face 1140 similar to camera rig 801. Each of the fourside faces 1130, 1132, 1134, 1136 of the camera rig 1101 includes anarray of cameras including a light field camera and a pair ofstereoscopic camera pair while the top face 1140 of camera rig includesat least one camera device 1125 similar to what has been shown anddiscussed with regard to camera rig 801. However the camera rig 1101further includes, in addition to the camera arrays on each of the fivefaces 1130, 1132, 1134, 1136 and 1140, a sixth bottom face 1142including at least one camera 1126 facing vertically downward, e.g.,towards the ground. In some such embodiments the bottom surface camera1126 facing vertically downwards and the top face camera 1125 facingvertically upwards are light field cameras. In some embodiments each ofthe cameras 1125 and 1126 are part of a corresponding stereoscopiccamera pair on the top and bottom faces 1140, 1142 of the camera rig1101.

While the stereoscopic cameras of the camera rigs 801 and 1101 are usedto capture stereoscopic imaging content, e.g., during an event, the useof light field cameras allows for scanning the scene area of interestand generate depth maps of various portions of the scene area capturedby the light field cameras (e.g., from the captured images correspondingto these portions of the scene of interest). In some embodiments thedepth maps of various portions of the scene area may be combined togenerate a composite depth map of the scene area. Such depth maps and/orcomposite depth map may, and in some embodiments are, provided to aplayback device for use in displaying stereoscopic imaging content andsimulating a 3D environment which can be experienced by the viewers.

While the stereoscopic cameras of the camera rigs discussed above areused to capture stereoscopic imaging content, e.g., during an event, theuse of light field cameras allows for scanning the scene area ofinterest and generate depth maps of various portions of the scene areacaptured by the light field cameras (from the captured imagescorresponding to these portions of the scene of interest). In someembodiments the depth maps of various portions of the scene area may becombined to generate a composite depth map of the scene area. Such depthmaps and/or composite depth map may, and in some embodiments are,provided to a playback device for use in displaying stereoscopic imagingcontent and simulating a 3D environment which can be experienced by theviewers.

The use of light field camera on combination with the stereoscopiccameras allows for environmental measurements and generation theenvironmental depth maps in real time, e.g., during an event being shot,thus obviating the need for deployment of environmental measurements tobe performed offline ahead in time prior to the start of an event, e.g.,a football game.

While the depth map generated from each image corresponds to a portionof the environment to be mapped, in some embodiments the depth mapsgenerated from individual images are processed, e.g., stitched together,to form a composite map of the complete environment scanned using thelight field cameras. Thus by using the light field cameras a relativelycomplete environmental map can be, and in some embodiments is generated.

In the case of light field cameras, an array of micro-lenses capturesenough information that one can refocus images after acquisition. It isalso possible to shift, after image capture, one's viewpoint within thesub-apertures of the main lens, effectively obtaining multiple views. Inthe case of a light field camera, depth cues from both defocus andcorrespondence are available simultaneously in a single capture. Thiscan be useful when attempting to fill in occluded information/sceneportions not captured by the stereoscopic cameras.

The depth maps generated from the light field camera outputs will becurrent and is likely to accurately measure changes in a stadium orother environment of interest for a particular event, e.g., a concert orgame to be captured by a stereoscopic camera. In addition, by measuringthe environment from the same location or near the location at which thestereoscopic camera are mounted, the environmental map, at least in someembodiments, accurately reflects the environment as it is likely to beperceived from the perspective of the stereoscopic cameras that are usedto capture the event.

In some embodiments images captured by the light field cameras can beprocessed and used to fill in for portions of the environment which arenot captured by a stereoscopic camera pair, e.g., because the positionand/or field of view of the stereoscopic camera pair may be slightlydifferent from that of the light field camera and/or due to anobstruction of view from the stereoscopic cameras. For example, when thelight field camera is facing rearward relative to the position of thestereoscopic pair it may capture a rear facing view not visible to aforward facing stereoscopic camera pair. In some embodiments output ofthe light field camera is provided to a playback device separately oralong with image data captured by the stereoscopic camera pairs. Theplayback device can use all or portions of the images captured by thelight field camera when display of a scene area not sufficientlycaptured by the stereoscopic camera pairs is to be displayed. Inaddition a portion of an image captured by the light field camera may beused to fill in a portion of the a stereoscopic image that was occludedfrom view from the position of the stereoscopic camera pair but which auser expects to be able to see when he or she shifts his or her head tothe left or right relative to the default viewing position correspondingto the location of the stereoscopic camera pair. For example, if a userleans to the left or right in an attempt to peer around a columnobstructing his/her view in some embodiments content from one or moreimages captured by the light field camera will be used to provide theimage content which was not visible to the stereoscopic camera pair butwhich is expected to be visible to the user from the shifted head potionthe user achieves during playback by leaning left or right.

Various exemplary camera rigs illustrated in FIGS. 1-9 may be equippedwith a variety of different cameras, e.g., normal cameras, stereoscopiccamera pairs, light field cameras etc. The exemplary camera rigs areused in various embodiments to capture, e.g., using the equippedcameras, environmental information, e.g., measurements and images, tosupport various applications in accordance with the features of thepresent invention.

FIG. 6 illustrates an exemplary system 6600 implemented in accordancewith some embodiments of the invention. The system 600 supportsenvironmental information measurement and capture including imagecapture, processing and delivery, e.g., imaging content, environmentalmodel and/or texture map delivery, to one or more customer devices,e.g., playback devices/content players, located at customer premises.The system 600 includes an exemplary imaging apparatus 604, astereoscopic imaging system 606, a processing system 608, acommunications network 650, and a plurality of customer premises 610, .. . , 612. The imaging apparatus 604 includes one or more light fieldcameras while stereoscopic imaging system 606 includes one or morestereoscopic cameras. In some embodiments the imaging apparatus 604 andthe stereoscopic imaging system 606 are included in an exemplary camerarig 602 which may be any of the camera rigs discussed earlier withregard to FIGS. 1-5. The camera rig 602 may include additional imagingand/or environmental measurement devices in addition to the light fieldcamera apparatus and the stereoscopic imaging system 606. The imagingapparatus 602 captures and processes imaging content in accordance withthe features of the invention. The communications network 650 may be,e.g., a hybrid fiber-coaxial (HFC) network, satellite network, and/orinternet.

The processing system 608 is configured to process imaging data receivedfrom the one or more light field cameras 604 and one or morestereoscopic cameras included in the stereoscopic imaging system 606, inaccordance with the invention. The processing performed by theprocessing system 608 includes, e.g., generating depth map of theenvironment of interest, generating 3D mesh models and UV maps,processing image content received from one or more camera devicespositioned at one or more location in the environment, e.g., encodingimage in one or more different formats, extract occluded image data inaccordance with the features of the present invention, and communicatingthe image content as well as environmental model information and UV mapsto one or more playback devices in accordance with the features of theinvention. In some embodiments the processing system 608 may include aserver with the server responding to requests for content and/oenvironmental information for use in rendering content, e.g., depth mapscorresponding to environment of interest, and/or 3D environmental meshmodels, UV maps and/or imaging content.

The playback devices may, and in some embodiments do, use suchinformation to simulate a 3D environment and render 3D image content.

The processing system 608 is configured to stream, e.g., transmit,imaging data and/or environmental information to one or more customerdevices, e.g., over the communications network 650. Via the network 650,the processing system 608 can send and/or exchange information with thedevices located at the customer premises 610, 612 as represented in thefigure by the link 609 traversing the communications network 650. Theimaging data and/or information may be encoded prior to delivery to oneor more playback devices.

Each customer premise 610, 612 may include a plurality ofdevices/players, which are used to decode and playback/display theimaging content, e.g., captured by stereoscopic cameras 606 and/or othercameras deployed in the system 600. The imaging content is normallyprocessed, e.g., formatted and/or encoded, prior to being communicatedto the playback devices by the processing system 608. The customerpremise 1 610 includes a decoding apparatus/playback device 622 coupledto a display device 620 while customer premise N 612 includes a decodingapparatus/playback device 626 coupled to a display device 624. In someembodiments the display devices 620, 624 are head mounted stereoscopicdisplay devices. In some embodiments the playback devices 622, 626receive and use the environmental model (also referred to as the 3D meshmodel), UV map and imaging content received from the processing system608 in rendering 3D imaging content and displaying the 3D imagingcontent to the user.

In various embodiments playback devices 622, 626 present the imagingcontent on the corresponding display devices 620, 624. The playbackdevices 622, 626 may be devices which are capable of decodingstereoscopic imaging content captured by stereoscopic camera, generateimaging content using the decoded content and rendering the imagingcontent, e.g., 3D image content, on the display devices 620, 624.

FIG. 7, which comprises the combination of FIGS. 7A, 7B, 7C and 7D showsmethod 700 of generating information, e.g., environmental model and UVmap information, and for generating and streaming content correspondingimages captured in the environment. The methods and apparatus cansupport the capture, processing and streaming of content in real timewhile an event is ongoing but can also be used for non-real time contentgeneration and streaming. While shown as a complete process from imagecapture to streaming, it should be appreciated that the model generationand processing of images to generate content in a format for streamingcan be performed by a separate system from the apparatus or system thatstreams the content. For example, once content is generated in one ormore formats it can be loaded onto one or more servers which receive andrespond to content requests, e.g., as described in FIG. 7. Thus whilethe steps of FIG. 7 will be explained using an example wherein a singlesystem performs the content generation and streaming related steps,different portions of the method 700 can, and in some embodiments are,performed by different devices.

The method 700 starts in step 702, e.g., with a content processing andstreaming system such as the one shown in FIG. 6 or any of the otherfigures of the application being powered on. In step 704, the systemreceives environmental model information indicating the shape of theenvironment, e.g., depth information from one or more sources. Theenvironmental model information may be depth information measuring theshape of an environment where images are to be captured. For example theinformation may be depth information measured at a sports stadium from adefault viewing position, e.g., a camera or seat position, from whichimages of an event which occurs in the environment will be captured.LIDAR may be used to make the environmental depth measurements. A lightfield camera may alternatively or in addition be used to capture depthinformation. Additional depth information may be obtained from opticalimages which are captured, e.g., by a camera rig placed at the defaultviewing location. Static model information of the shape of the stadiumor environment may also be used to generate a model of the environment.

The model information received in step 704, from one or more sourceswhich may include cameras of the camera rig shown in FIG. 4, operationproceeds to step 706 in which one or more models of the environment aregenerated. In some embodiments the models are mesh models with the meshmodeling one or more surfaces in the environment onto which textures,e.g., captured images, can be applied to generate images that a user canview during playback.

Operation proceeds from step 704 to step 706 in which one or moreenvironmental models are generated. The environment may be a stadium,theater, outdoor environment or any environment from which images may becaptured, e.g. by using one or more camera rigs. For example the camerarig in FIG. 4 or any of the other figures in this application maybe usedto capture images of the environment. Step 706 includes one or more orall of steps 708, 710 and 712.

In step 708 a model of the surfaces of the environment visible from thedefault viewing location, e.g., a first environmental model, isgenerated. The first model generated in step 708 maybe, and sometimesis, a mesh model generated from images and/or depth measurements madefrom the default viewing location, and/or static information about theshape of the environment. The first model, in some embodiments, modelsmodel surfaces in the environment which are visible from the defaultviewing location as a set of segments connected by nodes hence thereference to a “mesh model”. Each segment represents a surface ontowhich a texture, e.g., an image, maybe applied as part of generating animage to be displayed to a user of a playback device. Thus the firstmesh model represents the shape of the environment which is modeled aswould be perceived by someone located at the default viewing locationlooking out towards the modeled surfaces, e.g., walls, support columns,stage, etc, of the environment from the default viewing location.

As should be appreciated, from the default viewing location somesurfaces, e.g., of objects in the environment, may not be visible fromthe default viewing location. Such surfaces are referred to as “occludedobject or surfaces” because they are occluded from view when a personobserves the environment or a camera captures images of the environmentfrom the default location. Thus for purposes of explaining the inventionan occluded surface or object is to be considered an object or surfacewhich is not visible from the default viewing location while anon-occluded object or surface is to be considered an object or surfaceviewable from the default viewing location, e.g., as a user turns his orher head without shifting left or right, up or down, from the defaultviewing location. For example a rear side of a column or an objectbehind a column would be an occluded object if it is not visible fromthe default viewing location. An occluded image or occluded imageportion refers to an image of an occluded surface or object or a portionof an image of an occluded surface or object.

Fans at a sporting event or another event are accustomed to viewing theevent in the environment where the event is presented from a seat orother generally fixed viewing location. The position from which imagesof the environment from where content is captured may, and sometimesdoes, correspond to a seating position at a sporting event or otherevent.

While participants at events are accustomed to having assigned seats,they are also accustomed to being able to lean left or right in the seatand stand up or haunch down while at the event. Such an action normallyis limited in the distance from which a viewer may change his/her headposition and can be thought of as a shift or offset from the defaultviewing position, e.g., the head position of a fan if the fan was seatedat the event. Such movements are often done in an attempt to view aportion of the environment obscured, i.e., occluded, from view from thedefault viewing position. Unlike a simple head rotation or head tiltwhere a users head position may change but the location of the user'shead in the environment remains generally at the same location in theenvironment, an offset left or right, up or down, from the defaultviewing location is normally accompanied by an expectation that the fanor user will be able to see some portion of the environment which wasnot visible from the default viewing location.

The first environmental model which models surfaces visible from thedefault viewing location normally does not include segments and/or nodescorresponding to surfaces which are not visible from the default viewinglocation. This is because assuming a user stays at the default locationhe/she is not likely to view other surfaces and including such detailsabout portions of the environment which are not likely to be used wouldbe wasteful from a data storage and/or transmission perspective.

For devices which can not support changes in viewing location to anoffset location, the first environmental model should be adequate andwell suited for purposes of supporting playback of content captured inthe environment. However, for devices to support a user's ability ofchange his/her viewing position from the default position modelinformation regarding additional surfaces, e.g., surfaces which may beviewed if a user changes his viewing location, would be desirable. Forexample, it might be useful to model all or a portion of a column orwall or an inside surface of a box not visible from the default viewinglocation if by changing viewing location the user would expect to beable to see the surface. For example, by moving his head up a user mightexpect to be able to see into a box in front of him which the user couldsee the top of but not inside while at the default viewing location.Consider also that the user may expect to be able to see the back or aportion of a rear side of a column by leaning to the side and shiftinghis/her head location to the left or right of the default viewinglocation. The occluded surfaces a user is likely to be able to see byleaning left or right, or standing vs sitting, are likely to be arelatively small portion of the environment. However, to provide arealistic virtual reality experience it can be desirable to be able toaccurately present such occluded portions of an environment to a user inresponse to detecting a change in the location of the user's head.

In order to support the display of occluded objects and surfaces, insome embodiments auxiliary model information is generated in step 712modeling portions of the environment which are not visible from thefirst default viewing location. The auxiliary model information can bebased on images captured by a camera or distance measurement deviceportioned in the environment at a different location, e.g., a locationoffset from the default viewing location, and/or based on detailedenvironmental model information such as a detailed stadium model whichincludes information of surfaces which are not visible from the defaultviewing position. Many stadiums and/or other locations have beenmeasured in detail from a variety of angles to produce an accurate 3Dmodel of the environment and such models are a good source of theauxiliary model information when real time measurements are notpossible.

The auxiliary model information may be in the form of supplemental meshinformation which can be combined with the first mesh model to generatea more detailed model of the environment which includes segments andnodes corresponding to at least some portions/surfaces which are notvisible from the default viewing position. Depending on the embodiment,the auxiliary model information and the first environmental modelinformation can be combined to form a second environmental model as donein step 710. The second environmental model generated in step 710includes segments and nodes corresponding to surfaces visible from thedefault viewing location and also segments and nodes corresponding toportions of the environment which are not visible from the first defaultviewing position but which are visible from one or more other locations,e.g., a second location offset from the default viewing position.

While the second environmental model may be used even if occludedenvironmental content is not to be displayed, in cases where a device isnot capable of supporting multiple viewing locations, use of the firstmodel is more efficient since occluded image portions will not bedisplayed since the playback device lacks the capability to supportalternative viewing locations. This can be expected where a device isnot powerful enough to decode the primary content and the occludedcontent at the desired frame rate.

However, in cases where a user may shift his head position it can beuseful to send the second environmental model or a combination of thefirst environmental model and auxiliary information, so that a playbackdevice has reliable model information for the non-occluded imagesurfaces, also sometimes referred to herein as main or primaryenvironmental surfaces, as well as the occluded surfaces. As will bediscussed below, depending on playback device capabilities and/or theformat in which content is supplied to a playback device it might bedesirable to supply: i) the first environmental model, ii) the secondenvironmental model or ii) a combination of the first environmentalmodel and the auxiliary model information. If supplied with the firstenvironmental model and the auxiliary model information the receivingdevice can modify the first model based on the node and segmentinformation included in the auxiliary model information to generate thesecond model which includes information modeling non-occluded surfacesas well as at least some occluded surfaces.

The environmental models and/or model information generated in step 706is stored, e.g., in memory or other storage, in step 714 so that it isavailable to be provided to playback devices seeking to playback contentcorresponding to the modeled environment.

Operation proceeds from step 714 to step 716 in which a UV map or mapsare generated to be used with image content which is to be mapped ontothe first or second environmental models. Whether one or more UV mapsare generated depends on whether one or more frames, e.g., 2D images,are used to communicate textures to be applied to the environmentalmodel being used for playback.

The image content to be used as a texture is, in some embodiments,images captured from the default viewing location or a location offsetfrom the default viewing location which are to be used as textures whichcan be applied to the segments in one or more of the environmentalmodels. Each UV map indicates how to divide up a 2D image and mapsegments in the UV map to corresponding segments of the 3D mesh model ofthe environment. Normally 1 segment of a UV map corresponds to onesegment of the 3D environmental model.

The UV maps can be used for mono images, e.g., where a single image istransmitted for display to both the left and right eyes of a user. TheUV map can also be used to map left eye image content onto the 3D modelto generate a left eye image and to map a right eye image onto the 3Dmodel to generate a right eye image to be displayed. In such anembodiment the same UV map and 3D environmental model may be used forboth the left eye image and right eye images of a stereo image pair butwith the input image for the left eye differing from the input imageused to generate the right eye image. The manner in which 2D images areencoded and transmitted may vary depending on the particular embodiment.In some stereo embodiments left and right eye input images of theenvironment to be used as textures are transmitted in different frames.In other embodiments both left and right eye images are incorporatedinto a single frame, e.g., with one image on the top of the frame andthe other image below the first image or with the left and right eyeimages being communicated in an interlaced fashion, e.g., with odd linesof a frame being used for a left eye image and even lines being used fora right eye image. Before application of a UV map the playback devicemay separate the left and right eye images and then use the UV map todetermine how to apply the left eye input image to the 3D map andseparately use the UV map a second time to determine how to apply theright eye input image to the 3D mesh model to generate a right eye imagefor display.

Since the way in which left and right eye images may be packed into aframe for transmission of a stereo image pair to a playback device mayoccur in one of a variety of ways, Applicant will try to limit thediscussion of such options in the present application to the extentpossible.

Step 716 may include one, more or all of steps 718, 720 and 722. In step718 a first UV map for mapping portions of a 2D image, e.g., frame, ontothe first environmental map is generated. The portions of the frame tobe mapped in generating the first UV map may and normally does includeportions which are images of corresponding portions of the environmentvisible from the default viewing location. The first UV map may be, andin some embodiments is, for a 360 view of the environment but may be fora smaller portion of the environment.

In step 720 a second UV map is generated for mapping portions of a frameincluding image portions corresponding to portions of the environmentwhich are visible from the default viewing position and also includingimage portions corresponding to occluded portions of the environmentvisible from one or more locations other than the default viewinglocation. For example a majority of a frame may be dedicated tocommunicating an image of the environment captured from the defaultviewing location for use as a texture to be applied to non-occludedsegments of the modeled environment and another smaller, e.g., bottomportion of the frame dedicated to transmitting images corresponding tooccluded surfaces. The size of the segments of a frame used forcommunicating texture information for occluded objects may be, andsometimes are, smaller than the size of segments in the UV map used tomap non-occluded image portions to same size segment of the 3D model. Insuch a case the use of a small portion of the transmitted frame and UVmap to communicate a texture for a occluded portion of the environmentresults in the texture being of lower resolution than the non-occludedimage portions. However, the use of lower resolution textures foroccluded image portions than non-occluded portions allows for efficientuse of the available pixels in a frame since the non-occluded imageportions are much more likely to be viewed then the occluded imageportions.

By using a single frame to communicate both non-occluded image data anda small amount of occluded image data, the display of occluded imagecontent can be supported without significantly increasing the amount ofdata which needs to be transmitted as compared to the case where framesincluding only non-occluded image data are transmitted. To support thetransmission of occluded image data in a frame with non-occluded imagedata the non-occluded image data may be, and sometimes is, down sampledslightly, e.g., by 10 percent or less as compared to the case whereoccluded data is not packed into the frame with the non-occluded imagedata.

Rather than pack the occluded image data into a frame with non-occludedimage data, the occluded image data may be packed into a frametransmitted in an auxiliary content stream which is multiplexed with afirst content stream communicating frames of non-occluded image data,e.g., image portions corresponding to non-occluded portions of themodeled environment. In cases where an auxiliary frame is used tocommunicate image portions corresponding to occluded surfaces, in step722 an auxiliary UV map is generated for mapping portions of theAuxiliary frame to segments in the 3D mesh model which correspond tooccluded portions of the environment. The auxiliary map may andsometimes does include portions corresponding to different primaryframes. For example, different ⅕ portions of the auxiliary frame maycommunicate content to be used with different frames of a primarycontent stream, e.g., a set of M frames in the primary content stream.In such a case different portions of the auxiliary frame will be used atdifferent times based on the information included in the UV map so thatthe content in a single auxiliary frame can be combined with the contentin the primary frame for purposes of application to the mesh model togenerate an image corresponding to a frame time.

With the different mesh models having been generated in step 716 tosupport each of a first, second and third content streaming format,operation proceeds to step 724 in which the generated first UV map,second UV map and Auxiliary UV map are stored in memory, e.g., with themesh model information so that it can be supplied as needed to aplayback device requesting content.

Operation proceeds from step 724 to steps 726 and, via connecting node A727, to step 729. Step 726 marks the start of processing of imagecontent, e.g., one or more streams of images captured by a camera rig atthe default viewing potion in the environment. Thus step 726 in whichcaptured image content is received relates to the receipt ofnon-occluded image content. In step 726 the processing system receivesimage content captured by a first camera or a first camera pair,including the first camera and a second camera, located at the defaultviewing position. In the case where mono-images are being supported thefirst camera alone will be used. In cases where stereoscopic imagecapture is supported the first camera will be used to capture, e.g., aleft eye image, and the second camera will be used to capture, e.g., aright eye image.

Operation proceeds from step 726 to step 728 in which received imagecontent received from the first camera or first camera pair which is tobe encoded, is selected for inclusion in a first frame, e.g., a frameused to communicate image data intended to be used as textures forsegments of the 3D environmental mesh model. Step 728 may involvecropping of a received image or images and/or extraction of differentportions of a received image or images corresponding to differentportions of the environment to be included in a frame or frames to betransmitted to the playback device. The selection performed in step 728takes into consideration which portions of a received image or imagesare to be mapped onto the model with such portions being selected forinclusion in the frame to be transmitted.

Operation proceeds from step 728 to steps 732 and 734 via connectingnode B 730. In step 732, the image content selected in step 728 isformatted for encoding in accordance with a first format. This step mayinvolve further cropping, scaling and/or performing selective resolutionreduction and/or combining of content from different images forinclusion in a single frame. For example if left and right eye imagesare to be packed into a single frame step 732 will format the images sothat the images can be placed in the same frame which will often involvedownsampling. In the case of stereoscopic embodiments where the left andright eye images are to be transmitted as separate frames, step 732would output a pair of images forming to a stereoscopic frame pair asopposed to a single frame including both the left and right eye images.

Data 736 represents the image or image pair formatted for encoding inthe first format which output by step 732 and supplied as input to step741 in which encoding is performed on data 736 to generate an encodedframe or frames from the captured image images. Operation proceeds fromstep 741 to step 744 where the encoded frame or frames 746 in the firstformat are stored in a content store, e .g., memory, for possible futurestreaming in a first content stream using a first stream format and/or athird content stream using a third content stream format.

The received image content which was selected in step 728 is subjectedto processing to generate frames in a second format in some embodiments.The generation of the frames in the second format includes steps 734,740, 742 and uses occluded image data extracted in step 731 from analternative image source e.g., a second camera. The second format framesinclude occluded image data, e.g., portions of images of occludedobjects, in addition to image portions corresponding to non-occludedobjects which normally occupy the majority of the frame in the secondformat.

In step 734 an image or image pair is formatted for encoding inaccordance with the second frame format in which a single frame includesboth image portions corresponding to non-occluded portions of theenvironment and image portions corresponding to occluded portions of theenvironment. Step 734 may include downsampling of all or some portionsof the non-occluded image portions to make space in the frame or framesfor occluded image content. Thus in the second frame format a fewernumber of pixels may be dedicated to non-occluded environmental portionsthan when the first format is used assuming frames of the same size interms of pixel number are used for both the first and second frameformats. The image or image pair formatted for encoding generated instep 734 is represented in FIG. 7B by data 738 which serves as input tostep 740. Another input to step 740 is occluded image data 762 which isto be included in a frame in accordance with the second frame format.Generation of the occluded image data 762 will be discussed furtherbelow with regard to the processing of an image or images from a cameraat a location which is offset from the default viewing location and isthus capable of capturing images of at least some occluded objects.

In step 740 the formatted image data 738 is combined with the occludedimage data 762 to generated a frame or frames to be encoded inaccordance with the second format. In the case of mono where one frameis generated per frame period or stereo where left and right eye imagesare packed into a single frame, a single frame will be generated foreach playback frame period. In the case where different frames are to beused to communicate left and right eye images, a left eye image and aright eye image will be generated in step 740 to form a stereoscopicframe pair. In step 742 the formatted frame or frames generated in step742 are encoded. The encoded frame or frames 748 in the second frameformat, which include both non-occluded image portions and occludedimage portions, are stored in step 744, e.g., for use in responding tosubsequent content requests.

After UV map generation and storage, processing proceeds to step in step729 via connecting node A 727. Step 729 can be performed in parallelwith steps 732, 734 and is shown in parallel but could also be performedsequentially as long as the occluded image data is made available foruse in step 740.

In step 729 one or more images are received from an additional camera,e.g., a third camera located at a different location than said firstcamera and/or said first camera pair located at said default viewinglocation. The third camera captures in its images at least some portionsof the environment which are occluded from view from the default viewingand primary image capture location.

In step 731 image portions, e.g., occluded image data, corresponding toone or more portions of the environment which are not visible to thefirst camera or camera pair are extracted from the image or imagesreceived from the third camera. The extracted portions may correspond tothe back of a column, inside of a box, or some other surface not visiblefrom the default viewing location. The extracted occluded image portionsmay be non-contiguous image portions intended to be used as textures forsegments of the environmental which are occluded segments. Processing ofthe extracted occluded image segments proceeds from step 731 to step 752shown in FIG. 7C via connecting node C 751. In step 752 a check is madeto determine if there are any segments, occluded environmental segments,in the environmental model corresponding to areas not visible from thedefault viewing location for which occluded data is not available, e.g.,based on image data missing from the set of extracted occluded imagedata for one or more occluded segments of the environment which wasobtained in step 731. If there is no missing occluded image data theextracted occluded image data is complete and the extracted occludedimage data is supplied in step 761 to one or more other processingsteps, e.g., step 740, as occluded image data 762.

If in step 752 it is determined that occluded image data is missing forsome of the segments in the environmental model corresponding tooccluded portions of the environment, operation proceeds from step 752to step 754. In step 754 the environment and/or images of theenvironment included the occluded image portions are analyzed todetermine how best to generate textures from available image portions ofthe environment to fill the occluded segments where captured imagecontent is not available. In some embodiment's edge and/or objectdetection is used to determine if a segment for which image content isnot available is part of an object or surface for which image content isavailable. Edges in captured images of the environment are used toidentify surfaces and/or objects. In some embodiments rather than supplya texture, e.g., image portion, instructions are generated on how theplayback device should fill the occluded segment for which an imageportion is not available. This may, and in some embodiments does,include an instruction to average or copy one or more specific occludedimage portions corresponding to the same surface or object and use theresulting image content as the missing image portion to be applied as atexture to the segment for which image data was not captured. In otherembodiments in step 756 a texture, e.g., image portion, is generatedfrom other occluded image portions in step 756 and then included in theoccluded image data as if it was captured by a camera. In someembodiments the occluded image data is an occluded portion of theenvironment which was captured by a camera but does not corresponddirectly to the occluded segment for which an image was not captured,e.g., it may correspond to an adjacent occluded portion of theenvironment. Since such image data would not be available absent itbeing included in the occluded image data, by considering whether or notcontent is available for all occluded image segments of theenvironmental model allows the device processing the images from thethird camera to make better substitutions and/or provide instructionsthan a playback device which does not have direct access to the imagescaptured by the third camera could make.

Operation proceeds from step 756 to step 760 in which the generated fillinstructions and/or substitute image data generated for occluded imagesegments missing such data is combined with the extracted occluded imagedata. The generated set of occluded image data is then returned in step761 as occluded image data 762 for use in one or more other steps suchas steps 740 and step 735.

The processing performed in step 740 has already been described.Accordingly, the discussion will now turn to the processing of theoccluded image data in step 735 of FIG. 7B. In step 735 the occludedimage data 762 is formatted for encoding in a third format, e.g., theformat used for use in generating auxiliary frames including occludedimage portions, e.g., small portions of an image which correspond to anoccluded object that is not visible from the default viewing location.In step 735 occluded image data, to be used with one or more primaryframes is formatted, e.g. cropped, downsampled and combined into a setof pixel data which can be communicated in an auxiliary frame or portionof an auxiliary frame. The occluded image data may be a set of cutoutsfrom the third image which are to be used as textures for occludedportions of the environment. These small image portions may bedownsampled to reduce their size in terms of the number of pixels whichwill be used to transmit the occluded image data. With the occludedimage data having been selected and formatted in step 735 operationproceeds to step 737 where the occluded image data, e.g., snippets orsmall image portions corresponding to occluded surfaces, are arranged tobe included in one or more auxiliary frames. While in the case ofnon-occluded image data where images of adjacent portions of theenvironment are normally arranged next to each other, e.g., adjacent oneanother in a manner consistent with where they will be placed in theenvironment, the occluded image data which is a combination of imageportions corresponding to multiple objects may be, and sometimes is,arranged to maximize the use of the available space in an auxiliaryframe or frame portion. Thus in the case of auxiliary image datacorresponding to occluded objects the arrangement of the image portionsin the frame may be very different from the arrangement of the occludedimage portions in the actual environment with the UV map associated withthe auxiliary frame controlling where an image portion will be used inthe environment as a texture.

In the auxiliary frame, pixels corresponding to non-adjacent object inthe environment are sometimes placed next to each other with theunderstanding that the UV map corresponding the auxiliary frame whichcontrols where the image, e.g., texture, is applied in the modeledenvironment based on information provided by the UV map indicating howthe auxiliary frame portions should be mapped to the environmental modelsegments. Similarly, when occluded image content is included in a framegenerated in the second format, pixels corresponding to non-adjacentoccluded objects may be placed next to each other in the frame fortransmission purposes with the UV map that is used in conjunction withsuch a frame controlling to which segments of the environmental meshmodel the occluded image content will be applied as textures. Thus,unlike the primary or non-occluded image data where adjacent pixelsnormally correspond to adjacent segments in the mesh model, with regardto occluded image data adjacent pixels in a frame may, and often do,correspond to non-adjacent surfaces or objects, e.g., occluded objectsurfaces not visible from the default viewing location which are notadjacent one another. For example, an image of a rear portion of acolumn may be included next to, e.g., adjacent, an image of an inside ofa box in the occluded image data set used to form a frame or portion ofa frame in the case of auxiliary data even though the rear portion ofthe column and inside of the box are located at different non-adjacentlocations in the environment. Thus in the arrangement of occluded imagedata as part of step 737 adjacent image portions in the actualenvironment may be non-adjacent in the arranged set of occluded imagedata and images of occluded objects which are non-adjacent objects inthe environment may be arranged to be adjacent in the auxiliary frame oroccluded set of image data included in a frame of the second format.

Given that the occluded image portions tend to be a much smaller portionof the environment which is modeled than the non-occluded imageportions, when occluded image data is sent in an auxiliary frame theoccluded image data corresponding to multiple different primary framesmaybe included in a single auxiliary frame. In addition the auxiliaryframe size may be different, e.g., smaller than the primary frame size.The playback device can recover the occluded image data corresponding todifferent frames of a primary stream and use the recovered image datawith the corresponding frame of the primary, e.g., non-occluded imagecontent stream. Thus, by using a different, e.g., smaller, frame size interms of the number of pixels in a frame and/or by packing occludedimage data for multiple different non-occluded image frames into asingle auxiliary frame occluded image content can be provided withrelatively little overhead as compared to the amount of data required totransmit the primary content stream. In some embodiments the occludedimage content stream, e.g., auxiliary content stream, has a data rate of1/10th or less than the data rate of the primary non-occluded contentstream. The frame rate of the auxiliary data stream, in cases where theauxiliary frame included occluded image content to multiple primaryframes, will normally be a fraction of the frame rate of the primarystream. In some embodiments the frame rate of the auxiliary contentstream providing occluded image data to be used as textures is ⅕, 1/10or less than the frame rate of the primary content stream.

Once the occluded image content from one or more frames has beenarranged into a frame or frames in step 737, operation proceeds to step739 in which the auxiliary frame or frames are encoded, e.g.,compressed, for storage and/or transmission. The encoded auxiliary frameor frames 750 are then stored in step 744, e.g., for future streamingand/or processing.

Operation proceeds from step 744, in which the encoded content which canbe used to generate streams in various formats is stored, to step 772 ofFIG. 7D via connecting node E 771. In step 772 content setscorresponding to different stream formats are generated to facilitatestreaming of content in response to content requests. Step 772 includessteps 773, 777, and 783.

In step 773 a first content set 774 corresponding to a first contentstream format is generated. The generated first content set 774 includesthe first environmental model 775 which includes segments correspondingto the non-occluded segments of the environment, a first UV map 776 formapping frames of images corresponding to the environment to segments ofthe first environmental model 775 and encoded frames in the first format746 which can be sent as a content stream. The encoded frames 746 do notinclude occluded image content. Since the first set of content 774,includes content, map and model information which relates tonon-occluded image portions and does not include information foroccluded image portions, it is well suited for streaming to deviceswhich do not support the display of occluded image data, e.g., deviceswhich for processing power limitation reasons or data transmissionconstraint reasons will process and display non-occluded image portionsbut not occluded image portions.

In step 777 a second content set 778, e.g., corresponding to a secondcontent stream format, is generated, e.g., from the previously createdsecond environmental model 780, second UV map 781 and encoded frame orframes 748 in the second format. In the case of the second format framesthe frames include both non-occluded and occluded image content. Thesecond environmental model 780 includes segments corresponding tonon-occluded portions of the environment and occluded portions of theenvironment. The second UV map 781 includes information on how to mapnon-occluded image portions as well as occluded image portions to thesecond environmental model 780. Thus, in the second stream format casewhile a separate content stream with occluded image data is not sent, aportion of the frames which are sent are used to provide image portionscorresponding to occluded surfaces in the second environmental model780. While the second content stream format may require the same orapproximately the same amount of data to transmit as the first contentstream format, for devices which can not use the occluded image data itwould be better to receive a content stream in the first format sincemore of the transmitted frame is used for non-occluded, e.g., primary,image portions potentially allowing for better quality of the primaryimage portions visible from the default viewing position than would bepossible if less data, e.g., a smaller portion of each transmittedframe, was used for communicating non-occluded image content.

In step 783 a third content set 784 is generated. The third content set784 corresponds to a third content stream format. In accordance with thethird content stream format, a program or other content is streamedusing a multiplex of frames corresponding to non-occluded image contentin a primary content stream and frames used to provide occluded imagecontent in an auxiliary content stream. In some embodiment the eachauxiliary frame provides occluded image content to be used with multipledifferent primary content stream frames. Thus for each of M framestransmitted in the primary content stream, 1 frame is transmitted in theauxiliary content stream, where M is 1 or larger and often greater than5 or 10. This is because a much smaller amount of the availablebandwidth is dedicated to transmitting auxiliary frames, e.g., occludedimage content in some embodiments, rather than primary frames. In someembodiments the auxiliary content stream requires ⅕ or less data thanthe primary content stream and in many cases 1/10 or less data than theprimary, e.g., non-occluded image content stream.

Step 783 includes in some embodiments creating a content set whichincludes environmental model information 785 for use with the thirdstream format which includes model information for both non-occluded andoccluded portions of the environment. In some embodiments environmentalmodel information 785 includes first environmental model information 775and auxiliary model information 786 which can be used in combination togenerate an environmental model with both non-occluded and occludedsegments. In other embodiments the environmental module information 785includes the second environmental model 780 which includes segmentscorresponding to non-occluded and occluded portions of the environment.By including the second environmental module 780 in a content stream theplayback device does not have to combine the first environmental model775 and auxiliary model information 786 to create the environmentalmodel to be used and can simply rely on the second environmental module780 which already includes non-occluded and occluded segments.

In addition to third model information the third content set 784includes the first UV map 776 and a frame or frames 746 encoded in thefirst format, e.g., a format which includes non-occluded image contentbut not occluded image portions. The first UV map 776 can be used to mapportions of frames in the first format, i.e., non-occluded imageportions, onto segments of the second environmental model 780 whichcorrespond to non-occluded portions of the environment. In addition tothe non-occluded image content 746, in the third stream format occludedimage portions are communicated in auxiliary frames sent in an auxiliarycontent stream. Accordingly, the third content set 784 includes anauxiliary UV map 788 for mapping occluded image portions onto segmentsof the second environmental module 780 which correspond to occludedportions of the environment. Information about such occluded segments isincluded in the auxiliary model information 786 when the content streamincludes first environmental model information 775 and auxiliary moduleinformation 786 which allows the playback device to generate the secondenvironmental model 780.

With the content sets for the various streams generated and stored instep 772, they are available for streaming. As should be appreciated fora given program title, the corresponding content may be provided usingthe first stream format, the second stream format or the third streamformat by streaming content and related model/UV map information to aplayback device. While for a given program title multiple differentcontent streams maybe available, individual content streams maybeidentified by a specific content identifier. A playback device canrequest content by providing a content identifier, e.g., a program titleor number identifying a program for which one or more content setsexist, and allowing the device, e.g., server providing the content, toselect the content set 774, 778 or 784 to provide or the playback devicecan request content in a specific steam format by including in a contentrequest a content set or stream identifier corresponding to a specificone of the stored content sets 774, 778, 784.

In order to facilitate content requests in step 791 the system whichcreated and stored the content sets 774, 778, 784 publishes, e.g.,communicates to playback devices a list of the available content and/oravailable content streams. The list of available content may include aprogram title or program identifier while available stream informationmay include stream identifiers indentifying content corresponding to anindicated title in a specific content stream format. A playback devicereceiving the published available content information can use it torequest content, e.g., from a server which stores and streams thecontent sets 774, 778, 784 in response to requests. Devices withdifferent capabilities maybe and sometimes are streamed differentversions of the same title with, e .g., a device which does not supportthe display of occluded images being supplied with content set 774, asecond device with a single decoder and which supports the display ofoccluded images being supplied content set 778 and with a device thatsupports multiple decoders, e.g., a primary stream decoder and anauxiliary stream decoder, being supplied with content set 784.

With the list of available titles being distributed to playback devicesin step 791 to facilitate the making of content requests, operationproceeds to content serving routine 900 shown in FIG. 8 via go to step793.

Before discussing the content serving routine 900 shown in FIG. 8, abrief discussion of the various formats of the content streams which maybe served may be helpful to understanding the serving routine.

FIG. 9 shows a first stream format 1000 which is used to serve, e.g.,stream or download content which does not include occluded imageportions, e.g., content corresponding to images which can be viewed froma default viewing location. In accordance with the first format, anenvironmental model 1002, e.g., a mesh model of the environmentcorresponding to where the images to be streamed were captured isprovided along with a UV map 1004 for mapping portions of frames 1006,to the environmental model. In the first stream format the providedmodel 1002, UV map 1004 and image content 1006 correspond tonon-occluded portions of the environment, e.g., portions which arevisible from a default viewing position in the environment. The firststream format does not involve the communication of occluded imagecontent, e.g., portions of images which correspond to areas of theenvironment which are not visible from the default viewing position.

FIG. 10 shows a second stream format 1050 which is used to serve, e.g.,stream or download content, which includes non-occluded image portionsand occluded image portions in a frame, e.g., content corresponding toimages which can be viewed from a default viewing location as well ascontent corresponding to image which can not be viewed from the defaultviewing position but which are visible from a location offset from thedefault viewing position. In accordance with the second format, a secondenvironmental model 1052, e.g., a mesh model of the environmentcorresponding to where the images to be streamed were captured isprovided along with a UV map 1053. In accordance with the second formata frame or frames 1057 are encoded to include image content 1058corresponding to a non-occluded portion of the environment and imagecontent 1060 corresponding to occluded portions of the environment.While in FIG. 10 the upper portion of the frame 1057 is shown providingnon-occluded image content and the lower portion of the frame providingoccluded image content other configurations are possible, e.g., withoccluded image content being provided on both the top and bottom or leftand right portions of the frame. In some embodiments the non-occludedimage portions correspond to a contiguous area in the frame 1057 whileoccluded image content may be located at various locations, e.g., leftand right sides, bottom and top of the frame. Using a contiguous portionof the frame 1057 for non-occluded image data is not mandatory but mayfacilitate extraction and application of the non-occluded image portionsto the environmental model. In the second stream format, since thecommunicated frames include both non-occluded image contentcorresponding to the primary viewing area and occluded image content,the UV map 1053 includes a first portion 1054 indicating a mapping ofthe non-occluded image content to segments of the mesh model 1052 and asecond portion 1056 indicating a mapping of occluded image portions tosegments of the mesh model 1052.

FIG. 11 shows a third stream format 1200 which is used to serve, e.g.,stream or download content, which includes non-occluded image portionsbeing transmitted in frames of a main or primary content stream 1210 andoccluded image portions in an auxiliary stream 1250. In order facilitateuse the main stream frames 1212, 1220 providing non-occluded imagecontent and corresponding auxiliary stream frames 1252 providingoccluded image content, a playback device is the playback device, issupplied as part of the third stream format environmental mesh modelinformation in the form of a second mesh module 1203 which includessegments for non-occluded and occluded portions of the environment orinformation sufficient to construct such a model 1203 or information1202 sufficient to generate such a model. The information 1202sufficient to form such a model 1203 includes a mesh model 1242 of thesurfaces visible from the default viewing location, e.g., non-occludedsurfaces, and auxiliary mesh model information 1244 which providesinformation on how to add nodes and/or segments to the mesh model 1242to generate the second mesh model 1203.

In addition in the case of the third stream format since differentframes are used to supply primary non-occluded image content to be usedas textures and an auxiliary stream is used to supply frames whichprovide occluded image content to be used as textures, separate UV maps1274, 1275 are provided. UV map 1274 provides information on how to mapsegments of a transmitted frame onto segments, of the second mesh model1203, corresponding to non-occluded portions of the environment.Auxiliary UV map 1275 includes information on how to map a segment of atransmitted auxiliary frame onto segments of one or more non-occludedsegments of the mesh model 1203.

In the FIG. 11 embodiment for each M frames (1212, 1220) (1232, 1240) ofprimary non-occluded image content, one auxiliary frame 1252, 1260 withoccluded image content is generated and included in the auxiliarycontent stream 1250. For transmission purposes the frames of the mainstream 1210 corresponding to a program and the frames of the auxiliarystream 1250 corresponding to the same program may be and often aremultiplexed. Thus a content stream corresponding to a program mayinclude data corresponding to a main stream 1210 and another set of datacorresponding to auxiliary stream 1250. The content of the streams 1210,1250 may be, and often is, packetized with a multiplexer mixing thepackets together but with the streams to which they correspond beingidentifiable from the use of different stream ideas included in thepacket headers of different packets. On playback the content of thestreams 1210 and 1250 can be easily demultiplexed based on the streamidentification information communicated with the packets of each stream.

While M maybe 1 or greater it is normally at least 5 or larger since theamount of occluded image data is usually ⅕ or less than the amount ofprimary frame data for each primary frame. Since the main stream 1210includes M frames for each frame of the auxiliary stream 1250, the framerate of the main stream 1210 will be M times the frame rate of theauxiliary stream 1250. In cases where a single auxiliary frame includesauxiliary image content for multiple frames, the auxiliary contentcorresponding to different primary frame F₁ 1212 to F_(M) 1220 may bearranged in a predetermined manner into a single auxiliary frame AF₁1252. For example, as shown in FIG. 11, the auxiliary frame AF₁ 1252includes auxiliary image content for frame F₁ 1262 and each of the otherframes in the first set of M frames. Thus, auxiliary frame AF₁ includesauxiliary image data 1262 for primary frame F₁, image data for otherframes in the first set as represented by the use of three dots, andauxiliary data 1265 for primary frame F_(M) 1220.

In some embodiments playback device uses different decoders for decodingthe main and auxiliary encoded frames. While a hardware decoder orgraphics processor is often used in a playback device such as a cellphone to decode the frames 1212, 1220, 1232, 1240 of the primary contentstream providing non-occluded image data, the general purpose processorof the decoder is configured to decode the lower rate frames of theauxiliary stream 1250. In other cases where the hardware decoder is fastenough, it can be used in the playback device to switch between decodingencoded frames of the main stream 1210 and encoded frames 1250 of theauxiliary stream. Given that a slower decoder, e.g., slower than thedecoder used to decode frames of the main stream, maybe used to decodethe auxiliary frames in some embodiments the auxiliary frame 1252corresponding to a set of primary frames (1212, 1220) is sent to theplayback device prior to the corresponding set of primary frames to makesure that the auxiliary image content is available in decoded form atthe same time as the corresponding primary frame with which it is to beused.

FIG. 8 illustrates a content serving method 900 that maybe implementedby a server in accordance with one exemplary embodiment of the presentinvention. While operation of the exemplary content server will bedescribed in the context of an example where the server stores and canprovide, e.g., content in each of the first, second and third streamformats it should be appreciated that the sever need not support allformats and, depending on the embodiment may support a single one of thefirst, second or third formats or two of the three formats.

The method 900 starts in step 902, e.g., with a serving routine beingloaded and executed by a processor of the content server implementingthe method. Operation proceeds from start step 902 to monitoring step904 in which the server monitors for content requests, e.g., fromplayback systems and/or devices at one or more different customerpremises

In step 906 a request for content is received from a playback device.Depending on the information available to the playback device the devicemay indicate in the content request a particular content title leavingit up to the server to select which format to stream or the playbackdevice can identify a specific content stream, e.g., a content streamcorresponding to a user selected title and in a content stream formatselected by the playback device. From a device identifier included in orsent with the content request or from device capability informationprovided by the playback device, the server can determine if theplayback device can support the display of occluded data and if sowhether the first and/or second stream formats can be supported. Theserver may also know the data rate of the channel that can be used toserve the playback device from information provided by the playbackdevice or from monitoring of the network connection between the serverand playback device.

With a content request having been received, operation proceeds fromstep 906 to step 908. In step 908 the server determines theenvironmental model information, UV map or UV maps to be provided andwhich content stream or streams to supply to the playback device inresponse to the received request. The decision maybe and sometimes isbased on the capabilities of the playback device and/or the data rate tothe playback device that can be supported for content delivery.

Step 908 includes one or more substeps. In the FIG. 8 example, step 908begins with substep 910 in which a check is made to determine if thereceived content request includes a content identifier identify acontent in a specific stream format. For example is a stream identifiercorresponding to requested content was included in the content request,the answer to question 910 would be yes and operation would proceed fromstep 910 to 922. In step 922 the stream format to be used would be setto match the stream format of the specific content stream indicated inthe content request, e.g., the stream format to be used would be set tothe one of the first, second or third stream formats that matches theformat of the specifically requested stream. Operation proceeds fromstep 922 to step 930.

If the content request does not specifically identify a content streamhaving a particular format, operation proceeds from step 910 to step 912in which device capability is checked to determine if the playbackdevice from which the request was received supports processing anddisplay of occluded image data. Such a determination can be made basedon device capability information included in the request, e.g., devicecapability indicating support for the first, second and/or third streamformats and/or by looking up capability information based on theidentifier of the playback device included in the received request.

In step 912 if it is determined that the playback device requestingcontent does not support the processing and display of occluded imagedata operation proceeds to from step 912 to step 914. In step 914 it isdetermined that the first stream format, e.g., the format which does notsupply occluded image data, is to be used. Operation then proceeds fromstep 914 to step 924 in which the stream format to be used is set to thefirst stream format and then operation proceeds to step 930.

If in step 912 it is determined that the playback device which sent therequest supports the processing and display of occluded image content,operation proceeds from step 912 to step 916 in which the serverdetermines if the playback device processing of content streams in thethird stream format, e.g., the format in which a primary and auxiliarycontent stream are provided in a multiplexed stream, e.g., programstream including the primary stream, auxiliary stream and one or moreother streams such as audio streams.

Because the third format involves transmission of a primary frame streamand an auxiliary data stream providing occluded image content itnormally requires a high data transmission rate than the second datastream format to support content delivery. If in step 916 it isdetermined that the third stream format can be supported by the playbackdevice operation proceeds to step 918. In step 918, a check is made todetermine if the data rate required for a content stream in the thirdformat available for use in delivering content to the requestingplayback device. This can be determined based on data rate informationreceived from another device, determined by the server and/or reportedfrom the playback device requesting the content. If in step 918 it isdetermined that the data rate required for the third stream format canbe supported, operation proceeds to step 926 where the stream format tobe used is set to the third stream format prior to operation proceedingto step 930. If in step 918 it is determined that the data rate requiredto support the third stream format is not available to the playbackdevice from which the request was received, operation proceeds to step920. Also, if in step 920 it was determined that the playback devicerequesting content does not support the third stream format, operationproceeds from step 916 to step 920 in which it is determined that thesecond stream format is to be used. Operation proceeds from step 920 tostep 928. In step 928 the stream format to be used is set to the secondstream format and then operation proceeds to step 930.

In step 930 a content set corresponding to the determined stream formatto be used and requested content is accessed, e.g., retrieved frommemory. If the determined stream format to be used is the first streamformat the content set in FIG. 9 is accessed. If the determined streamformat to be used is the second stream format the content set in FIG. 10is accessed. If the determined stream format to be used is the thirdstream format the content set in FIG. 11 is accessed.

Operation proceeds from step 930 to step 932 in which the UV mapinformation, e.g., UV map or maps, from the accessed content set is sentto the playback device. Operation proceeds from step 932 to step 934 inwhich the environmental model information, e.g., environmental model ormodels, is sent to the playback device. The playback device can use thetransmitted UV map information and model information to render imagesusing content, e.g., frames, transmitted from the accessed set ofcontent corresponding to the requested program.

From step 934 operation proceeds to step 936 in which framescorresponding to the requested content are transmitted in accordancewith the determined stream format. In the case of the third streamformat, in step 938 which is preformed when the third stream format isused, frames of main image data will be multiplexed with auxiliaryframes providing occluded image data.

Operation is shown proceeding form step 936 to step 904 to show thatmonitoring for requests occurs on an ongoing basis. As requests arereceived, they are processed and the content is supplied to therequesting playback device.

As a result of serving different devices and their content requests, thesteps of figure 900 maybe implemented in response to a content from afirst device which does not support use of occluded image data in whichcase in step 936 the first device would be stream a content streamcorresponding to the first stream format. A second device which supportsthe second content stream format and use of occluded image data but notthe third format would be responded to differently when the same contentis requested but the stream format is not specified. For example if thesecond device requested the same program as the first device it would beprovided the program content stream which complies with the secondstream format. If a third device supporting a high data rate and thethird content stream format requested the same program it would beresponded to with the server providing the requested program content inthe third content stream format. Thus the server may and sometimes doessupply content corresponding to the same program in different contentstream formats to different devices at the same time depending on thedevices capabilities and/or the data rates that can be used to delivercontent to the devices. The processing described with regard to FIGS. 7and 8 is performed under control of a processor in some embodiments.Accordingly, in some embodiments the image processing system includes aprocessor configured to control the processing system to implement thesteps shown in FIGS. 7 and 8. The transmission and receiving steps areperformed via the interfaces (which include transmitters and receivers)of the playback devices.

FIG. 12 illustrates an exemplary processing system 1700 in accordancewith the features of the invention. The processing system 1700 can beused to implement one or more steps of the method of flowcharts 700and/or 900. The processing system 1700 includes encoding capability thatcan be used to encode and stream imaging content in a variety offormats. The exemplary processing system 1700 may be used as theprocessing system 608 of system 600.

The processing system 1700 may be, and in some embodiments is, used togenerate environmental models, UV maps, and image content that can beused for 3D image rendering, storage, and transmission and/or contentoutput in accordance with the features of the invention. The processingsystem 1700 may also include the ability to decode and display processedand/or encoded image data, e.g., to an operator.

The system 1700 includes a display 1702, input device 1704, input/output(I/O) interface 1706, a multiplexer 1707, a processor 1708, networkinterface 1710 and a memory 1712. The various components of the system1700 are coupled together via bus 1709 which allows for data to becommunicated between the components of the system 1700.

The memory 1712 includes various routines and modules which whenexecuted by the processor 1708 control the system 1700 to implement thecomposite environmental depth map generation, environmental depth mapreconciling, encoding, storage, and streaming/transmission and/or outputoperations in accordance with the invention.

The display device 1702 may be, and in some embodiments is, a touchscreen, used to display images, video, information regarding theconfiguration of the processing system 1700, and/or indicate status ofthe processing being performed on the processing device. In the casewhere the display device 602 is a touch screen, the display device 602serves as an additional input device and/or as an alternative to theseparate input device, e.g., buttons, 1706. The input device 1704 maybe, and in some embodiments is, e.g., keypad, touch screen, or similardevice that may be used for inputting information, data and/orinstructions.

Via the I/O interface 1706 the processing system 1700 may be coupled toexternal devices and exchange information and signaling with suchexternal devices, e.g., such as the camera rig 801 and/or other camerarigs shown in the figures and/or other external cameras. The I/Ointerface 1606 includes a transmitter and a receiver. In someembodiments via the I/O interface 1706 the processing system 1700receives images captured by various cameras, e.g., stereoscopic camerapairs and/or light field cameras, which may be part of a camera rig suchas camera rig 801. In some embodiments the cameras providing images tothe system 1700 are positioned at different locations and thus provideimage of portions of an environment of interest captured from differentlocations.

The multiplexer 1707 is configured to multiplex various frames includingimage content to generate the multiplexed content stream 1744. In someembodiments the multiplexer 1707 is configured to multiplex a frame infirst format (e.g., 1732′) and an auxiliary frame (e.g., 1736′). In someembodiments the multiplexer 1707 is configured, as part of beingconfigured to multiplex the first frame and the auxiliary frame, toincorporate the auxiliary frame in the multiplexed content stream beforethe first frame such that a device receiving the multiplexed contentstream will receive said auxiliary frame before the first frame.

The processor 1708, e.g., a CPU, executes routines 1714 and uses thevarious modules to control the system 1700 to operate in accordance withthe invention. The processor 1708 is responsible for controlling theoverall general operation of the system 1700, e.g., by controlling theprocessing system to perform a set of operations in accordance with theinvention, e.g., such as discussed in detail in the flowcharts 700 and900. In various embodiments the processor 1708 is configured to performfunctions that have been discussed as being performed by the processingsystem 1700.

The network interface 1710 allows the processing system 1700 to be ableto receive and/or communicate information to an external device over acommunications network, e.g., such as communications network 105. Thenetwork interface 1710 includes a transmitter 1740 and a receiver 1742.The transmitter 1740 allows the processing system 1700 to transmit,e.g., broadcast and/or unicast, encoded image content to variouscustomer devices. In some embodiments the processing system 1700transmits different portions of a scene, e.g., 180 degree front portion,left rear portion, right rear portion etc., to customer devices via thetransmitter 1740. Furthermore, in some embodiments via the transmitter1740 the processing system 1700 also transmits an environmental depthmap, one or more 3D environmental mesh models, one or more UV maps,and/or image content, e.g., stereoscopic imaging content, to individualcustomer devices. In some embodiments the transmitter 1740 is configuredto transmit the multiplexed content stream 1744 including the firstframe in a primary content stream and the auxiliary frame in anauxiliary content stream, to one or more playback devices.

The memory 1712 includes various modules and routines, which whenexecuted by the processor 1708 control the operation of the system 1700in accordance with the invention. The processor 1708, e.g., a CPU,executes control routines and uses data/information stored in memory1712 to control the system 1700 to operate in accordance with theinvention and implement one or more steps of the method of flowchart ofFIGS. 7 and 8.

The memory 1712 includes control routines 1714, a primary image encoder1716, an auxiliary encoder 1717, streaming controller 1720, a 3D meshmodel generation and update module 1722, a UV map generation and updatemodule 1722, received images 1723 of environment of interest captured byone or more cameras, generated frames of image content including frameof frames in first format 1732, frame of frames in second format 1734,and auxiliary frame of frames in third format 1736, encoded imagecontent including encoded frame of frames in first format 1732′, encodedframe of frames in second format 1734′, and encoded auxiliary frame offrames in third format 1736′, multiplexed content stream 1744, generatedenvironmental mesh models 1746, generated UV map(s) 1752.

In some embodiments the modules are, implemented as software modules. Inother embodiments the modules are implemented outside the memory 1712 inhardware, e.g., as individual circuits with each module beingimplemented as a circuit for performing the function to which the modulecorresponds. In still other embodiments the modules are implementedusing a combination of software and hardware. In the embodiments whereone or more modules are implemented as software modules or routines, themodules and/or routines are executed by the processor 1708 to controlthe system 1700 to operate in accordance with the invention andimplement one or more operations discussed with regard to flowcharts 700and/or 900.

The control routines 1714 include device control routines andcommunications routines to control the operation of the processingsystem 1700. The primary encoder 1716 may, and in some embodiments do,include a plurality of encoders configured to encode received imagecontent, e.g., stereoscopic images of a scene and/or one or more sceneportions, in accordance with the features of the invention. In someembodiments the primary encoder 1716 is configured to encode frame orframes in the first format and frame or frames encoded in the secondformat. The encoded frame or frames in the first and second format 1732′and 1734′ are output of the primary encoder 1716 which are stored in thememory for streaming to customer devices, e.g., playback devices. Insome embodiments the auxiliary encoder 1717 is configured to encodeframe or frames in the third format to output the encoded frame orframes in the third format 1736′. The encoded content can be streamed toone or multiple different devices via the network interface 1710 in someembodiments.

The streaming controller 1720 is configured to control streaming ofencoded content for delivering the encoded image content to one or morecustomer playback devices, e.g., over the communications network 605. Invarious embodiments the streaming controller 1720 is further configuredto communicate, e.g., control transmission via the transmitter 1740, oneor more environmental mesh models and UV maps to one or more customerplayback devices, e.g., via the network interface 1710.

The 3D environmental mesh model generation and update module 1722 isconfigured to generate the various types of 3D environmental mesh modelsin accordance with the features of the present invention as discussed indetail with regard to flowchart 700. In some embodiments the generated3D mesh model(s) 1746, which is the output of the 3D environmental meshmodel generation and update module 1722, includes one or more 3D meshmodels generated by module 1722 including a first environmental meshmodel 1747, a second environmental mesh model 1748 and auxiliaryenvironmental mesh model information 1750. The UV map generation andupdate module 1722 is configured to generate UV maps in accordance withthe features of the invention to be used in wrapping frames onto acorresponding 3D environmental mesh model. The generated UV map(s) 1752,which is the output of the UV map generation module 1722, includes afirst UV map 1754, second UV map 1756 and auxiliary UV map 1758. In someembodiments the modules are configured to perform the functionscorresponding to various steps discussed in FIGS. 7 and 8.

Received images 1723 includes images received from one or more cameras,e.g., such as those included in the rig 801 or other cameras deployed tocapture images in an environment of interest. The received images 1723includes a first image 1726 corresponding to a portion of anenvironment, said first image including a non-occluded image portioncorresponding to a portion of the environment visible from a firstlocation, a second image 1728 corresponding to a portion of anenvironment from the second camera, said second image including a secondnon-occluded image portion corresponding to the portion of theenvironment visible from the first location and an additional image 1730of the environment including at least a first occluded image portioncorresponding to a portion of the environment occluded from view fromsaid first location

In some embodiments the processor 1708 is configured to control theimage processing system 1700 to implement the steps shown in FIGS. 7 and8. The transmission and receiving steps are performed via the interfaces(which include transmitters and receivers) of the playback devices. Insome embodiments the processor 1708 is configured to control the system1700 to receive (e.g., via interface 1706 or via receiver 1742) a firstimage corresponding to a portion of an environment, said first imageincluding a non-occluded image portion corresponding to a portion of theenvironment visible from a first location, receive (e.g., via interface1706 or via receiver 1742) an additional image of the environmentincluding at least a first occluded image portion corresponding to aportion of the environment occluded from view from said first location,generate a first frame including image content from said non-occludedimage portion of said first image and image content from said firstoccluded image portion of the additional image; and store (e.g., inmemory 1712) said first frame in a storage device or transmit said firstframe to another device.

FIG. 13, which comprises the combination of FIGS. 13A and 13B,illustrates the steps of a method 1300 of operating a playback device inone exemplary embodiment. In some embodiments the playback and renderingsystem 1900 is used to implement the steps of the method of flowchart1300. In the FIG. 13 exemplary embodiment the playback device receivesinformation, e.g., available content stream information, indicatingvarious content streams available for delivery that the playback devicemay request.

The method of flowchart 1300 begins in start step 1302 with a playbackdevice, e.g., such as a game console and display or head mounted displayassembly, being powered on and set to begin receiving, storing andprocessing 3D related image data and information, e.g., framesrepresenting texture information, environmental model information and/orUV maps to be used in rendering images. Operation proceeds from startstep 1302 to step 1304 in which information communicating a list ofavailable content streams optionally including list of streamscorresponding to different stream formats is received, e.g., from theprocessing system, and stored, e.g., in memory. The list of contentincludes, e.g., information indicating various content items, e.g.,titles, available that the playback device can request to be streamedfor playback. Each title may be available in a variety of streamformats. While stream formats information, e.g., the list of streamscorresponding to different stream formats, may be communicated by theprocessing system to the playback device in some embodiments it may notalways be the case.

In step 1306 the playback device which normally monitors for user inputdetects user selection of content, e.g., user selection of a title.Operation proceeds from step 1306 to step 1308 in which the playbackdevice determines if streams in different stream formats are availableto select from for the user selected content, e.g., the playback devicedetermines whether user selected content is available in more than onestream format. The playback device may be able to make the determinationbased on stream formats information if such information was received bythe playback device. If such information is available to the playbackdevice and the device determines that user selected content in availablein multiple stream formats the operation proceeds to step 1310.

First referring to the processing along the path of step 1310, in step1310 the playback device determines, based on device capabilityinformation and/or current data rate supportable by the playback device,a content stream to be requested, e.g., from the different availablestreams in different stream formats. As part of step 1310 in someembodiments the playback device performs one or more of 1312 through1322. In step 1312 the device checks if processing and display ofoccluded image data is supported by the playback device, e.g., based ondevice capability, current processing power and/or hardware and/orsoftware availability or other constraints. If it is determined that forwhatever reasons at that point in time processing and display ofoccluded image data is not supported or desired by the playback devicethe operation proceeds from step 1312 to step 1314 where the playbackdevice determines and decides that the first stream format is to be usedand thus a stream supporting the first stream format for the userselected content is to be requested.

If however in step 1312 it is determined that processing and display ofoccluded image data is supported by the playback device, the operationproceeds from step 1312 to step 1316 wherein it is determined whetherthe playback device supports processing content streams in third streamformat, e.g., third stream format being a format supporting a multiplexof sub-streams with a first sub-stream providing content correspondingto main, e.g., non-occluded data, and another sub-stream providingcontent corresponding to occluded data. If it is determined that theplayback device does not support third stream format the processingproceeds from step 1316 to step 1318 wherein it is determined that thesecond stream format is to be used. If in step 1316 it is determinedthat the playback device supports the third stream format the processingproceeds to step 1320 wherein the device checks if the data raterequired for receiving and processing content stream in the third streamformat can be supported at the given time. If it is determined that suchdata rate can be supported the processing proceeds from step 1320 tostep 1322, otherwise, the operation proceeds to step 1318. In step 1322it is determined that the third stream format is to be used therebyconcluding the determination step 1310.

Following the determination regarding the content stream to be requestedthe operation proceeds from step 1310 to step 1326 via connecting node A1324. In step 1326 the playback device transmits a request for userselected content in determined format, e.g., first, second or thirdstream format, as determined in accordance with step 1310. Operationproceeds from step 1326 to step 1330.

Returning to step 1308. If in step 1308 it is determined that streams indifferent stream formats are mot available to select from for the userselected content and/or if the playback device is unable to make thedetermination of step 1308 due to unavailability of stream formatinformation the operation proceeds from step 1308 to step 1328 viaconnecting node B 1309. In step 1328 the playback device transmits arequest for user selected content, optionally with a device typeidentifier and/or device capability information to facilitate serverselection of content stream to supply. Operation proceeds from step 1328to step 1330.

In step 1330 a response to the request for content is received by theplayback device, e.g., from the processing server acknowledging that therequest for user selected content was received. Operation proceeds fromstep 1330 to step 1332. In step 1332 the playback device determines theformat of the content stream to be received, e.g., corresponding tofirst stream format, second stream format, or third stream format. Upondetermining the stream type to be received the playback deviceconfigures its hardware, software and/or firmware to allow the playbackdevice to receive, decode and process the content stream.

Depending on the type of content stream format in which the contentstream is to be received the operation proceeds along one of the threepaths corresponding to steps 1333, 1334 and 1335 as illustrated. If theuser selected content is to be received in a content stream in the firststream format, the operation proceeds from step 1332 to step 1333wherein the first stream format playback routine is invoked which isillustrated in FIG. 14 discussed in detail below. If the user selectedcontent is to be received in a content stream in the second streamformat, the operation proceeds from step 1332 to step 1334 where thesecond stream format playback routine is called which is illustrated inFIG. 15. If the user selected content is to be received in a contentstream in the third stream format, the operation proceeds from step 1332to step 1335 where the third stream format playback routine is calledwhich is illustrated in FIG. 16.

FIG. 14 illustrates the steps of an exemplary first stream formatplayback routine 1400 which is called and implemented by the playbackdevice of the present invention as part of performing the method offlowchart 1300. The processing of routine 1400 begins in step 1402 withthe playback device calling, e.g., executing, the first stream formatplayback routine 1400. Operation proceeds from step 1402 to step 1404where the playback device sets the user's initial head position defaultforward viewing position at a default viewing location. The defaultviewing location may correspond to the initial seating position of theuser facing forward from where the user would be able to view a scene inthe environment of interest, e.g., including the main scene area visiblefrom the default viewing position.

Operation proceeds from step 1404 to step 1406. In step 1406 theplayback device receives a first environmental model, e.g., 3D meshmodel of surfaces which can be seen from default viewing location.Operation proceeds from step 1406 to step 1408 where a first UV map,e.g., texture map including information indicating mapping of portionsof an image communicated in a frame to the first environmental model isreceived, with the portions of the image being portions which arevisible from a default viewing location.

Operation proceeds from step 1408 to step 1410. In step 1410 theplayback device receives content from a first content streamcommunicating frames, e.g., of image content, in the first format. Nextin step 1412 the playback device decodes a received frame to recover animage or images, e.g., recovering a left eye image and a right eye imagein the case of receiving a frame of stereoscopic image pair.

Operation proceeds from step 1412 to step 1414 where user's current headposition, e.g., head position at the given time, is determined.Operation proceeds from step 1414 to step 1416 in which the playbackdevice renders an image or images of the environment. In variousembodiments rendering an image includes using the received UV map to mapportions of the recovered left and right eye images onto portions of thefirst environmental model. Next in step 1418 a portion of the renderedimage or images of the environment which would be visible from aposition in the modeled environment to which the user's current headposition corresponds is displayed to the user, e.g., on a displaydevice. Thus in this manner images corresponding to the portions of theenvironment which are visible from the default viewing location arerendered and displayed to the user. Operation may proceed from step 1418back to step 1410 as indicated by the loopback and various steps may berepeated for additional received content frames.

FIG. 15, which comprises the combination of FIGS. 15A and 15B,illustrates the steps of an exemplary second stream format playbackroutine 1500 which is called and implemented by the playback device insome embodiments of the present invention as part of performing themethod of flowchart 1300. The second stream format playback routine 1500is called in some embodiments if the user selected content is to bereceived in a content stream in the second stream format. The processingof routine 1500 begins in step 1502 with the playback device calling,e.g., executing, the second stream format playback routine 1500.Operation proceeds from step 1502 to step 1504 where the playback devicesets the user's initial head position default forward viewing positionat the default viewing location. Operation proceeds from step 1504 tostep 1506. In step 1506 the playback device receives a secondenvironmental model, e.g., 3D mesh model of surfaces which can be seenfrom default viewing location and at least some portions which can notbe seen from the default viewing location in the environment. It shouldbe noted that in comparison the first 3D mesh model, the secondenvironmental model provides additional data corresponding to portionswhich can not be seen from the default viewing location. Operationproceeds from step 1506 to step 1508 where the playback device receivesa second UV map including information indicating mapping of portions ofan image communicated in a frame having the second format to the secondenvironmental model is received, with the portions of the image beingimages of portions of the environment visible from the default viewinglocation and portions of the environment not visible from the defaultviewing location but one or more other location, e.g., offset from thedefault viewing location. In some embodiments the second UV mapoptionally includes information on how to generate image content for oneor more segments not visible from the default location in the absence ofimage content being supplied for such occluded segments.

Operation proceeds from step 1508 to step 1510. In step 1510 theplayback device receives content from a second content streamcommunicating frames in the second format which includes image portionscorresponding to at least some occluded portions of the environment.Next in step 1512 the playback device decodes a received frame of thesecond content stream to recover an image or images, e.g., recovering aleft eye image and a right eye image in the case of receiving a frame ofstereoscopic image pair.

Operation proceeds from step 1512 to step 1516 via connecting node A1514. In step 1516 the playback device renders an image or images of theenvironment, e.g., using the second UV to map portions of the recoveredleft eye image onto portions of the second environmental model and usingthe second UV to map portions of the recovered right eye image ontoportions of the second environmental model.

Operation proceeds from step 1516 to step 1518. In step 1518 user'scurrent head position, e.g., head position at the given time, isdetermined. Next in step 1520 the playback device determines if theuser's current head position indicates a shift, e.g., offset, inlocation from the default viewing location in the environment. Forexample based on determined current head position of the user it isdetermined if the user has moved left, right, up or down rather thansimply rotating or tilting his/her head. This can be determined based ondetected changes in the user's current position relative to the user'sinitial head position. Based on the determination of step 1520 theoperation proceeds to one of the steps 1522 or 1524.

If in step 1520 it is determined that the user's current head positiondoes not indicate a shift in location from the default viewing locationin the environment the operation proceeds from step 1520 to step 1522where the playback device displays a portion of the rendered image orimages of the environment which would be visible from the defaultviewing location in the modeled environment taking into considerationthe user's current head position, e.g., viewing direction at the defaultlocation. Thus if no shift/offset is detected in user's current headposition it can be safely considered that the user has not moved fromthe default viewing location and accordingly image content correspondingto portions visible from the default viewing location are displayed. Insome embodiments as part of step 1522 the playback device performs step1523 where image portions corresponding to environmental segmentsviewable from the default viewing location are displayed withoutdisplaying some environmental segments occluded from view from thedefault viewing location. Operation proceeds from step 1522 back to step1510 via connecting node B 1526 as illustrated and various steps may berepeated for additional received content frames.

On the other hand if in step 1520 it is determined that the user'scurrent head position indicates a shift in location from the defaultviewing location in the environment the operation proceeds from step1520 to step 1524 where the playback device displays a portion of therendered image or images of the environment which would be visible fromthe offset location in the modeled environment offset from the defaultviewing location taking into consideration the user's current headposition, e.g., viewing direction at the location which is differentfrom the default viewing location. In some embodiments as part of step1524 the playback device performs step 1525 where image portionscorresponding to environmental segments viewable from the defaultviewing location are displayed along with at least some environmentalsegments occluded from view from the default viewing location. Thus if ashift/offset in user's current head position is detected, the playbackdevice is configured to display image content corresponding to at leastsome environmental portions occluded from view from the default viewinglocation in addition to portions visible from the default viewinglocation. Operation proceeds from step 1524 back to step 1510 viaconnecting node B 1526 as illustrated and various steps may be repeatedfor additional received content frames.

FIG. 16, which comprises the combination of FIGS. 16A, 16B, 16C and 16D,illustrates the steps of an exemplary third stream format playbackroutine 1600 which is called and implemented by the playback device insome embodiments as part of performing the method of flowchart 1300. Thethird stream format playback routine 1600 is called in some embodimentsif the user selected content is to be received in a content stream inthe third stream format. The processing of routine 1600 begins in step1602 with the playback device calling, e.g., executing, the third streamformat playback routine 1600.

Operation proceeds from step 1602 to step 1604 where the playback devicesets the user's initial head position default forward viewing positionat the default viewing location. Operation proceeds from step 1604 tostep 1606. In step 1606 the playback device receives environmental modelinformation for surfaces visible from the default viewing location andat least some surfaces not visible from the default location, e.g.,primary 3D mesh model of surfaces which can be seen from default viewinglocation and supplemental, e.g., auxiliary, mesh model information whichcan be combined with primary mesh model to generate a thirdenvironmental model. Alternatively in some embodiments the playbackdevice receives third environmental model including nodes and segmentscorresponding to environmental surfaces visible from default viewinglocation and nodes and segments corresponding to some surfaces which arenot visible from default viewing location but visible from a differentviewing location offset from the default location.

Operation proceeds from step 1606 to optional step 1608 in someembodiments. Step 1608 is performed in embodiments where the playbackdevice receives the first, e.g., primary, mesh model along withauxiliary environmental information corresponding to at least somesurfaces not visible from the default location. In step 1608 theplayback device generates the third mesh model of the environment bycombining the primary mesh model and the auxiliary environmental modelinformation corresponding to at least some surfaces not visible from thedefault location. The generated third mesh model includes nodes andsegments corresponding to the environmental surfaces visible from thedefault viewing location and some surfaces not visible from the defaultlocation but visible from a different location such as a location offsetfrom the default viewing location. Thus irrespective of which form theenvironmental information is received, the playback device gets theenvironmental model information for surfaces visible from the defaultviewing location and at least some surfaces not visible from the defaultlocation.

Operation proceeds from step 1608 (or from step 1606 in embodimentswhere step 1608 is skipped) to step 1612 via connecting node A 1610. Instep 1612 the playback device receives primary UV map and auxiliary UVmap. As part of step 1612 in some embodiments steps 1614 and 1616 areperformed. In step 1614 the playback device receives a primary UV mapincluding information indicating mapping of portions of an imagecommunicated in a frame of a main/primary content stream, e.g., firstcontent stream, to segments of the third environmental model whichcorrespond to surfaces visible from the default viewing location . Instep 1616 the playback device receives an auxiliary UV map includinginformation indicating mapping of image portion communicated in anauxiliary frame to segments of the third environmental model whichcorrespond to surfaces not visible from the default viewing location butvisible from one or more other locations, e.g., offset from the defaultlocation. In some embodiments the auxiliary UV map provides mappinginformation for occluded objects for one or more different framesincluded in the primary content stream.

Operation proceeds from step 1612 to step 1620. In step 1620 theplayback device receives a multiplexed content stream corresponding tothe third stream format providing both the primary frame(s) andauxiliary frame(s) communicating image content. As part of receivingimage content frames in step 1620 the playback device receives a frameor set of frames from primary content stream in sub-step 1622 andreceives a frame or frames from the auxiliary content stream in sub-step1624. Operation proceeds from step 1620 to step 1630 via connecting nodeB 1626.

In step 1630 the playback device demultiplexes the multiplexed contentstream and outputs primary and auxiliary frames for further processing,e.g., decoding. To facilitate a better understanding the output of thedemultiplex operation performed on the multiplexed content stream isshown as two data sets 1632 and 1642 coming out of the demultiplex step1630. The first set of frames 1632, e.g., primary frames, includesframes 1 to M of image content corresponding to portions visible fromthe default viewing position while the auxiliary frame set 1642 includesauxiliary frame or frames of image content corresponding to portions notvisible from the default viewing position. The two data sets 1632 and1642 serve as input to two different decoders in some embodiments. Thefirst set of frames 1632 is supplied to a primary decoder which may be adedicated hardware decoder configured to decode and recover datacommunicated by primary frames corresponding to portions visible fromthe default viewing position. As shown in step 1634 the primary decoderis used to decode image content corresponding to primary frames. Theoutput of the decoding performed by the primary decoder in step 1634 isthe set of decoded primary frames 1636 including decoded frame 1 1638 todecoded frame M 1640. Operation proceeds from step 1634 to step 1656 inwhich the decoded primary frames 1636 serves as an input.

In some embodiments the auxiliary frame or frames 1642 is supplied to asecondary/auxiliary decoder which the playback device uses to decode andrecover data communicated by auxiliary frames of image contentcorresponding to portions not visible from the default viewing positionas shown in step 1644, e.g., providing occluded image data. The outputof the decoding performed by the auxiliary decoder in step 1644 is thedecoded auxiliary frame or frames 1646, e.g., frame or frames providingimages of occluded segments of the environment for one or more primaryframes. In the example of FIG. 16 the decoded auxiliary frame or frames1646 includes a single decoded auxiliary (aux) frame 1648 includingoccluded image content corresponding to multiple primary frames. Thedecoded aux frame 1648 in this example is in such a format that it packsoccluded image content for frame 1 1650, occluded image content forframe 2 1652, . . . , and occluded image content for frame M 1654.Operation proceeds from step 1634 to step 1656 in which the decoded auxframe 1646 serves as an input.

While in figure the use of a primary decoder 1920 and secondary decoder1921 are shown in FIG. 16C, in some embodiments the primary decoder 1920is used on a time shared basis to decode both the primary and theauxiliary frame. In such embodiments the primary decoder 1920 and thusalso serves as the ancillary frame decoder 1921. For such embodiments itis useful to have the primary and auxiliary frames to be of the samesize to minimize the amount of decoder reconfiguration required tosupport switching between decoding the primary frames and decoding theauxiliary frames.

In some other embodiments the primary decoder 1920 is a differentdecoder, e.g., a decoder implemented as a separate processor orprocessor core from the auxiliary decoder. In some embodiments theprimary decoder is implemented as a dedicated hardware video decoderwhile the auxiliary decoder is implemented on a general purposeprocessor, e.g., CPU, of the playback device such as the cell phone. Theauxiliary decoder maybe slower and/or have less processing power thanthe primary decoder. In some embodiments, e.g., when the auxiliarydecoder is less powerful than the primacy decoder, the auxiliary framesare of smaller size and/or include less pixels than the primary decoder.This facilitates auxiliary decoder implementation using software or aless powerful decoder than is used for the primary decoder. This isparticular helpful where the playback device is a cell phone includingdecoder circuitry or a graphics processor which can be used as theprimary decoder and also includes a general purpose processor which canbe configured, e.g., under stored instructions, to operate as theauxiliary decoder.

In step 1656 the playback device renders an image or images of theenvironment, e.g., with the playback device using the primary UV map tomap image portions included in the recovered decoded primary frame orframes onto the portions of the third environmental model and use theauxiliary UV map to map image portions included in the recovered decodedaux frame corresponding to a mapped primary frame onto the portions ofthe third environmental model which are normally occluded.

Operation proceeds from step 1656 to step 1660 via connecting node C1658. In step 1660 user's current head position, e.g., head position atthe given time, is determined. Next in step 1662 the playback devicedetermines if the user's current head position indicates a shift, e.g.,offset, in location from the default viewing location in theenvironment, e.g., determine if the user has moved left, right, up ordown rather than simply rotating or tilting his/her head. Based on thedetermination of step 1662 the operation proceeds to one of the steps1664 or 1668.

If in step 1662 it is determined that the user's current head positiondoes not indicate a shift in location from the default viewing locationin the environment the operation proceeds from step 1662 to step 1664where the playback device displays to the user a portion of the renderedimage or images of the environment which would be visible from thedefault viewing location in the modeled environment taking intoconsideration the user's current head position, e.g., viewing directionat the default location. In some embodiments as part of step 1664 theplayback device performs step 1666 where image portions corresponding toenvironmental segments viewable from the default viewing location aredisplayed without displaying some environmental segments occluded fromview from the default viewing location. Operation proceeds from step1664 back to step 1620 via connecting node D 1672 as illustrated andvarious steps may be repeated for additional received content frames.

If in step 1662 it is determined that the user's current head positionindicates a shift in location from the default viewing location in theenvironment the operation proceeds from step 1662 to step 1668 where theplayback device displays a portion of the rendered image or images ofthe environment which would be visible from the offset location in themodeled environment offset from the default viewing location taking intoconsideration the user's current head position, e.g., viewing directionat the location which is different from the default viewing location. Insome embodiments as part of step 1668 the playback device performs step1670 where image portions corresponding to environmental segmentsviewable from the default viewing location are displayed along with atleast some environmental segments occluded from view from the defaultviewing location. Thus if a shift/offset in user's current head positionis detected, the playback device is configured to display image contentcorresponding to at least some environmental portions occluded from viewfrom the default viewing location in addition to portions visible fromthe default viewing location. Operation proceeds from step 1668 back tostep 1620 via connecting node D 1672 as illustrated and various stepsmay be repeated for additional received content frames.

The processing described with regard to FIG. 13 is performed undercontrol of a playback device processor. Accordingly, in some embodimentsthe playback device includes a processor configured to control theplayback device to implement the steps shown in FIG. 13. Thetransmission and receiving steps are performed via the interfaces (whichinclude transmitters and receivers) of the playback devices.

In some embodiments the playback device includes instructions which,when executed by a processor of the playback device, control theplayback device to implemented the steps shown in FIG. 13. Separateprocessor executable code can be and sometimes is included for each ofthe steps shown in FIG. 13. In other embodiments a circuit is includedin the playback device for each of the individual steps shown in FIG.13.

FIG. 17 illustrates an exemplary 3D mesh model 2000 that may be used invarious embodiments with a plurality of nodes illustrated as the pointof intersection of lines used to divide the 3D model into segments. Notethat the model of FIG. 17 is shown in 3D space and can be expressed as aset of [X,Y,Z] coordinates defining the location of the nodes in themesh in 3D space assuming the shape of the segments is known or therules for interconnecting the nodes is known or defined in the 3D model.In some embodiments the segments are predetermined to have the samenumber of sides with each node connecting to a predetermined number ofadjacent nodes by straight lines. In the FIG. 17 example the top portionof the model 2000 is a set of triangular segments while the sideportions are formed by a plurality of four sided segments. Such aconfiguration, e.g., top portion being formed of 3 sided segments and aside portion formed by 4 sided segments may be included in theinformation forming part of the 3D model or predetermined. Suchinformation is provided to the customer rendering and playback devicesalong with or as part of the mesh model information.

FIG. 18 shows an exemplary UV map 2002 which may be used in mapping aframe in what is sometimes referred to as 2D UV space to the 3D model2000 shown in FIG. 18. Note that the UV map 2002 includes the samenumber of nodes and segments as in the 3D model 2000 with a one to onemapping relationship. Frames which provide what is sometimes referred toas texture, but which normally include content of images captured fromthe vantage point of a camera rig in a real environment, at a locationcorresponding to the position [0, 0, 0] within the 3D model 2000 of thesimulated environment, may be applied, e.g., wrapped, on to the 3D model2000 in accordance with the map 2002 as part of an image renderingoperation.

In FIGS. 17 and 18, exemplary node P which is shown as a dot foremphasis, like each of the other mesh nodes, appears in both the UV map2002 and the 3D model 2000. Note that the node P[X, Y, Z] corresponds tothe node P[U,V], where X, Y, Z specify the position of node P in X, Y, Zspace and U,V specify the location of the corresponding node P in thetwo dimensional space. Each U,V pair represents the X, Y of a singlepixel of the 2D image texture, e.g., a frame. Surrounding pixels aremapped from the 2D frame to the 3D mesh during the rendering process byinterpolating between nearby U,V pairs.

FIG. 19 illustrates an exemplary playback device, e.g., system, 1900that can be used to receive, decode and display the content streamed byone or more sub-systems of the system 600 of FIG. 12, e.g., such as theprocessing system 608/1700. The exemplary rendering and playback system1900 may be used as any of the rendering and playback devices shown inFIG. 12. In various embodiments the playback system 1900 is used toperform the various steps illustrated in flowcharts shown in FIGS.13-16.

The rendering and playback system 1900 in some embodiments includeand/or coupled to 3D head mounted display 1905. The system 1900 includesthe ability to decode the received encoded image data and generate 3Dimage content for display to the customer. The playback system 1900 insome embodiments is located at a customer premise location such as ahome or office but may be located at an image capture site as well. Theplayback system 1900 can perform reception, decoding, rendering, displayand/or other operations in accordance with the invention.

The playback system 1900 includes a display 1902, a display deviceinterface 1903, a user input interface device 1904, input/output (I/O)interface 1906, a demultiplexer 1907, a processor 1908, networkinterface 1910 and a memory 1912. The various components of the playbacksystem 1900 are coupled together via bus 1909 which allows for data tobe communicated between the components of the system 1900.

While in some embodiments display 1902 is included as an optionalelement as illustrated using the dashed box, in some embodiments anexternal display device 1905, e.g., a head mounted stereoscopic displaydevice, can be coupled to the playback system 1900 via the displaydevice interface 1903. The head mounted display 1902 maybe implementedusing the OCULUS RIFT™ VR (virtual reality) headset which may includethe head mounted display 1902. Other head mounted displays may also beused. The image content is presented on the display device of system1900, e.g., with left and right eyes of a user being presented withdifferent images in the case of stereoscopic content. By displayingdifferent images to the left and right eyes on a single screen, e.g., ondifferent portions of the single screen to different eyes, a singledisplay can be used to display left and right eye images which will beperceived separately by the viewer's left and right eyes. While variousembodiments contemplate a head mounted display to be used in system1900, the methods and system can also be used with non-head mounteddisplays which can support 3D image.

The operator of the playback system 1900 may control one or moreparameters and/or provide input via user input device 1904. The inputdevice 1904 may be, and in some embodiments is, e.g., keypad, touchscreen, or similar device that may be used for inputting information,data and/or instructions.

Via the I/O interface 1906 the playback system 1900 may be coupled toexternal devices and exchange information and signaling with suchexternal devices. In some embodiments via the I/O interface 1906 theplayback system 1900 receives images captured by various cameras, e.g.,stereoscopic camera pairs and/or other cameras, receive 3D mesh modelsand UV maps.

The demultiplexer 1907 is configured to demultiplex multiplexed framescorresponding to image content communicated in a multiplexed contentstream, e.g., from the processing system 1700. In some embodiments thedemultiplexer 1907 is configured to demultiplex a primary content streamand an auxiliary content stream which are multiplexed.

The processor 1908, e.g., a CPU, executes routines 1914 and uses thevarious modules to control the system 1900 to operate in accordance withthe invention. The processor 1908 is responsible for controlling theoverall general operation of the system 1900, e.g., by controlling thesystem 1900 to perform various operations in accordance with thefeatures of the present invention. In various embodiments the processor1908 is configured to perform functions that have been discussed asbeing performed by the rendering and playback system 1900.

The network interface 1910 includes a transmitter 1911 and a receiver1913 which allows the playback system 1900 to be able to receive and/orcommunicate information to an external device over a communicationsnetwork, e.g., such as communications network 650. In some embodimentsthe playback system 1900 receives, e.g., via the interface 1910, encodedframes including image content 1924, 3D mesh model(s) 1928, UV map(s)1930, from the processing system 1700 over the communications network650.

The memory 1912 includes various modules, e.g., routines, which whenexecuted by the processor 1908 control the playback system 1900 toperform operations in accordance with the invention. The memory 1912includes control routines 1914, a request generation module 1916, areceived information processing module 1917, a head position and/orviewing angle determination module 1918, a primary decoder 1920, anauxiliary decoder 1921, a 3D image renderer 1922 also referred to as a3D image generation module, received encoded image content 1924,received list of available programs 1926, received 3D mesh model(s)1928, received UV map(s) 1930, decoded image content 1932, generatedimage content 1934 and device capability information 1934.

The control routines 1914 include device control routines andcommunications routines to control the operation of the system 1900. Therequest generation module 1916 is configured to generate request forcontent, e.g., upon user selection of an item for playback. The receivedinformation processing module 1917 is configured to process information,e.g., image content, audio data, environmental models, UV maps etc.,received by the system 1900, e.g., via the receiver of interface 1906and/or 1910, to provide the received information to an appropriateelement of the system 1900 for use in rendering and playback. The headposition and/or viewing angle determination module 1918 is configured todetermine a user's current head position, e.g., position of the headmounted display, in accordance with the features of the presentinvention.

The primary decoder 1920 is configured to decode frames received in aprimary content stream, e.g., encoded frames in first format (includedin the received encoded data 1924) received from the processing system1700 to produce decoded image data corresponding to frames in firstformat which is stored in the memory and included in decoded data 1932.The auxiliary decoder 1921 is configured to decode auxiliary frame orframes in the third format (included in the received encoded data 1924)to produce decoded image data corresponding to frames in the thirdformat included in decoded data 1932. In some embodiments the auxiliarydecoder 1921 is slower than the primary decoder 1920.

In some other embodiments the primary decoder 1920 is used to decode areceived first frame including both non-occluded image contentcorresponding to a portion of an environment visible from a firstlocation in the environment and occluded image content corresponding toa portion of the environment which is not visible from said firstlocation. In such embodiments the decoded image data 1932 includesdecoded frame or frames including both non-occluded image content andoccluded image content. The In various embodiments the encoded imagecontent is decoded prior to image rendering.

The 3D image renderer 1922 uses decoded image data 1932 to generate 3Dimage content in accordance with the features of the invention fordisplay to the user on the display 1902 and/or the display device 1905.In some embodiments the 3D image renderer 1922 is configured to render,using a 3D mesh model at least some of received image content. In someembodiments the 3D image renderer 1922 is further configured to use afirst UV map to determine how to wrap an image included in receivedimage content onto the first 3D mesh model. The generated image content1934 is the output of the 3D image rendering module 1922.

The received 3D environmental mesh model(s) 1928 may include a firstmesh model, a second mesh model 1928 and auxiliary mesh modeinformation. The received UV map(s) 1930 include a first UV map and/or asecond UV map and/or an auxiliary UV map. The received encoded imagecontent 1924 includes, e.g., frames of left and right eye image pairs ofnon occluded image content corresponding to a portion of an environmentof interest visible from a first location in the environment and one ormore auxiliary frames including occluded image content corresponding toa portion of the environment which is not visible from the firstlocation. In some embodiments the system 1900 receives a first frameincluding both non-occluded image content corresponding to a portion ofthe environment visible from a first location in the environment andoccluded image content corresponding to a portion of the environmentwhich is not visible from said first location. Thus in some embodimentsthe received encoded image content 1924 includes the frame includingboth non-occluded image content and occluded image content.

As discussed earlier the first UV map indicates a mapping betweenportions of a frames including non-occluded image content to segments ofa first model of the environment which correspond to portions of theenvironment visible from said first location while the auxiliary UV mapindicates a mapping between portions of the auxiliary frame includingoccluded image content to segments of the first model of the environmentwhich correspond to portions of said environment which are not visiblefrom said first location. When a received from includes bothnon-occluded image content and occluded image content, a second UV mapis used which indicates a mapping between portions of the frame andsegments of a corresponding environmental model, e.g., second 3D meshmodel. The device capability information 1936 includes informationregarding the processing and/or display capability of system 1900indicating whether the playback device 1900 is capable of supporting thedisplay of occluded image content corresponding to portions of theenvironment which are not visible from the first location and/orindicating content stream formats that can be supported by the playbackdevice 1900. In some embodiments device capability information 1936 isin the form of an identifier that can be used, e.g., by the processingsystem 1700, to determine capabilities of the playback device 1900.

In some embodiments some of the modules are implemented, e.g., ascircuits, within the processor 1908 with other modules beingimplemented, e.g., as circuits, external to and coupled to theprocessor. Alternatively, rather than being implemented as circuits, allor some of the modules may be implemented in software and stored in thememory of the playback device 1900 with the modules controllingoperation of the playback device 1900 to implement the functionscorresponding to the modules when the modules are executed by aprocessor, e.g., processor 1908. In still other embodiments, variousmodules are implemented as a combination of hardware and software, e.g.,with a circuit external to the processor 1908 providing input to theprocessor 1908 which then under software control operates to perform aportion of a module's function.

While shown in FIG. 19 example to be included in the memory 1912, themodules shown included in the memory 1912 can, and in some embodimentsare, implemented fully in hardware within the processor 1908, e.g., asindividual circuits. In other embodiments some of the elements areimplemented, e.g., as circuits, within the processor 1908 with otherelements being implemented, e.g., as circuits, external to and coupledto the processor 1908. As should be appreciated the level of integrationof modules on the processor and/or with some modules being external tothe processor may be one of design choice.

While shown in the FIG. 19 embodiment as a single processor 1908, e.g.,computer, within device 1900, it should be appreciated that processor1908 may be implemented as one or more processors, e.g., computers. Whenimplemented in software, the modules include code, which when executedby the processor 1908, configure the processor, e.g., computer, toimplement the function corresponding to the module. In some embodiments,processor 1908 is configured to implement each of the modules shown inmemory 1912 in FIG. 19 example. In embodiments where the modules arestored in memory 1912, the memory 1912 is a computer program product,the computer program product comprising a computer readable medium,e.g., a non-transitory computer readable medium, comprising code, e.g.,individual code for each module, for causing at least one computer,e.g., processor 1908, to implement the functions to which the modulescorrespond.

As should be appreciated, the modules illustrated in FIG. 19 controland/or configure the system 1900 or elements therein respectively suchas the processor 1908 to perform the functions of corresponding steps ofthe methods of the present invention, e.g., such as those illustratedand/or described in the flowcharts 1300, 1400, 1500 and 1600.

In one exemplary embodiment the processor 1908 is configured to controlthe playback device 1900 to receive, e.g., via receiver 1913, a firstframe including non-occluded image content corresponding to a portion ofan environment visible from a first location in the environment andoccluded image content corresponding to a portion of the environmentwhich is not visible from said first location; detect a head position ofa user, and output to a display an image of portions of the environmentas a function of the detected head position.

Various additional exemplary embodiments illustrating different aspectsand features of the invention will now be described.

A method embodiment 1 of operating an image processing system, themethod comprising: receiving a first image corresponding to a portion ofan environment, said first image including a non-occluded image portioncorresponding to a portion of the environment visible from a firstlocation; receiving an additional image of the environment including atleast a first occluded image portion corresponding to a portion of theenvironment occluded from view from said first location; generating afirst frame including image content from said non-occluded image portionof said first image and image content from said first occluded imageportion of the additional image; and storing said first frame in astorage device or transmitting said first frame to another device.

A method embodiment 2 of operating an image processing system, themethod embodiment 1 further comprising: generating a UV map to be usedfor mapping portions of said first frame to segments of an environmentalmodel.

A method embodiment 3 of operating an image processing system, themethod embodiment 1 further comprising: generating an environmentalmodel including segments corresponding to non-occluded surfaces of saidenvironment and segments corresponding to occluded surfaces of saidenvironment.

A method embodiment 4 of operating an image processing system, themethod embodiment 3 wherein said UV map maps image content in said firstframe corresponding to said non-occluded image portion of said firstframe to a first segment of said environmental model which correspondsto a portion of said environment which is visible from said firstlocation.

A method embodiment 5 of operating an image processing system, themethod embodiment 4 wherein said UV map further maps image content insaid first frame corresponding to said first occluded image portion to asecond segment of said environmental model which corresponds to aportion of said environment which is not visible from said firstlocation.

A method embodiment 6 of operating an image processing system, themethod embodiment 1 wherein receiving a first image includes receivingsaid first image from a first camera in said environment; and receivingan additional image of the environment includes receiving saidadditional image from a camera at a location in said environment offsetfrom a location at which said first camera is positioned in saidenvironment.

A method embodiment 7 of operating an image processing system, themethod embodiment 6 wherein said first camera is a camera of astereoscopic camera pair including said first camera and said secondcamera, said stereoscopic camera pair being positioned at said firstlocation, the method further comprising: receiving a second imagecorresponding to a portion of an environment from the second camera,said second image including a second non-occluded image portioncorresponding to the portion of the environment visible from the firstlocation.

A method embodiment 8 of operating an image processing system, themethod embodiment 7 further comprising: including in said first frame atleast a portion of said second image.

A method embodiment 9 of operating an image processing system, themethod embodiment 1 wherein said first frame is in a second frameformat, the method further comprising: generating a frame in a firstframe format, said frame in the first format including image contentfrom said non-occluded image portion of said first frame and no imagecontent corresponding to an occluded portion of the environment; andstoring said frame in the first format in the storage device.

A method embodiment 10 of operating an image processing system, themethod embodiment 9 wherein the first frame and the frame in the firstformat correspond to a first program, the method further comprising:receiving a first request for content corresponding to the first programfrom a first playback device which supports the display of occludedimage content; and sending said first frame in the second format to thefirst device in response to said first request for content.

A method embodiment 11 of operating an image processing system, themethod embodiment 10 further comprising: determining from devicecapability information, prior to sending the first frame in the secondformat, that the first playback device supports the display of occludedimage content.

A method embodiment 12 of operating an image processing system, themethod embodiment 10 further comprising: receiving a second request forcontent corresponding to the first program from a second playback devicewhich does not support the display of occluded image content; andsending said frame in the first format to the second playback device inresponse to said second request for content.

A method embodiment 13 of operating an image processing system, themethod embodiment 12 further comprising: determining from devicecapability information corresponding to the second playback device,prior to sending the frame in the first format, that the second playbackdevice does not support the display of occluded image content.

A method embodiment 14 of operating an image processing system, themethod embodiment 13 wherein said first frame in the second format andthe frame in the first format are the same size and include the samenumber of pixels.

A method embodiment 15 of operating an image processing system, themethod embodiment 14 further comprising: transmitting to the secondplayback device an environmental model which does not include segmentscorresponding to portions of the environment occluded from view from thefirst location; and transmitting to the second playback device a UV mapto be used for mapping portions of the frame in the first format to theenvironmental model which does not include segments corresponding toportions of the environment occluded from view from the first location.

Another exemplary embodiment includes a non-transitory computer readablemedium for use in a system, said non-transitory computer readable mediumincluding computer executable instructions which, when executed by acomputer, control the system to: receive a first image corresponding toa portion of an environment, said first image including a non-occludedimage portion corresponding to a portion of the environment visible froma first location; receive an additional image of the environmentincluding at least a first occluded image portion corresponding to aportion of the environment occluded from view from said first location;generate a first frame including image content from said non-occludedimage portion of said first image and image content from said firstoccluded image portion of the additional image; and store said firstframe in a storage device or transmit said first frame to anotherdevice.

An image processing system embodiment 1 comprising: a processorconfigured to control said image processing system to: receive a firstimage corresponding to a portion of an environment, said first imageincluding a non-occluded image portion corresponding to a portion of theenvironment visible from a first location; receive an additional imageof the environment including at least a first occluded image portioncorresponding to a portion of the environment occluded from view fromsaid first location; generate a first frame including image content fromsaid non-occluded image portion of said first image and image contentfrom said first occluded image portion of the additional image; andstore said first frame in a storage device or transmit said first frameto another device.

An image processing system embodiment 2, the image processing systemembodiment 1 wherein said processor is further configured to controlsaid image processing system to generate a UV map to be used for mappingportions of said first frame to segments of an environmental model.

An image processing system embodiment 3, the image processing systemembodiment 1 wherein said processor is further configured to controlsaid image processing system to generate an environmental modelincluding segments corresponding to non-occluded surfaces of saidenvironment and segments corresponding to occluded surfaces of saidenvironment.

An image processing system embodiment 4, the image processing systemembodiment 3 wherein said UV map maps image content in said first framecorresponding to said non-occluded image portion of said first frame toa first segment of said environmental model which corresponds to aportion of said environment which is visible from said first location.

An image processing system embodiment 5, the image processing systemembodiment 4 wherein said UV map further maps image content in saidfirst frame corresponding to said first occluded image portion to asecond segment of said environmental model which corresponds to aportion of said environment which is not visible from said firstlocation.

An image processing system embodiment 6, the image processing systemembodiment 1 wherein said first image is received from a first camera insaid environment; and wherein said additional image is received from acamera at a location in said environment offset from a location at whichsaid first camera is positioned in said environment.

An image processing system embodiment 7, the image processing systemembodiment 6 wherein said first camera is a camera of a stereoscopiccamera pair including said first camera and said second camera, saidstereoscopic camera pair being positioned at said first location whereinsaid processor is further configured to control said image processingsystem to receive a second image corresponding to a portion of anenvironment from the second camera, said second image including a secondnon-occluded image portion corresponding to the portion of theenvironment visible from the first location.

An image processing system embodiment 8, the image processing systemembodiment 7 wherein said processor is further configured to controlsaid image processing system to include in said first frame at least aportion of said second image.

An image processing system embodiment 9, the image processing systemembodiment 1 wherein said first frame is in a second frame format; andwherein said processor is further configured to control said imageprocessing system to: generate a frame in a first frame format, saidframe in the first format including image content from said non-occludedimage portion of said first frame and no image content corresponding toan occluded portion of the environment; and store said frame in thefirst format in the storage device.

An image processing system embodiment 10, the image processing systemembodiment 9 wherein the first frame and the frame in the first formatcorrespond to a first program; wherein said processor is furtherconfigured to control said image processing system to: receive a firstrequest for content corresponding to the first program from a firstplayback device which supports the display of occluded image content;and send said first frame in the second format to the first device inresponse to said first request for content.

An image processing system embodiment 11, the image processing systemembodiment 10 wherein said processor is further configured to controlsaid image processing system to determine from device capabilityinformation, prior to sending the first frame in the second format, thatthe first playback device supports the display of occluded imagecontent.

An image processing system embodiment 12, the image processing systemembodiment 10 wherein said processor is further configured to controlsaid image processing system to: receive a second request for contentcorresponding to the first program from a second playback device whichdoes not support the display of occluded image content; and send saidframe in the first format to the second playback device in response tosaid second request for content.

An image processing system embodiment 13, the image processing systemembodiment 12 wherein said processor is further configured to controlsaid image processing system to determine from device capabilityinformation corresponding to the second playback device, prior tosending the frame in the first format, that the second playback devicedoes not support the display of occluded image content.

An image processing system embodiment 14, the image processing systemembodiment 13 wherein said first frame in the second format and theframe in the first format are the same size and include the same numberof pixels.

An image processing system embodiment 15, the image processing systemembodiment 14 wherein said processor is further configured to controlsaid image processing system to: transmit to the second playback devicean environmental model which does not include segments corresponding toportions of the environment occluded from view from the first location;and transmit to the second playback device a UV map to be used formapping portions of the frame in the first format to the environmentalmodel which does not include segments corresponding to portions of theenvironment occluded from view from the first location.

A method embodiment 16 of operating an image processing system, themethod embodiment 16 comprising: receiving a first image correspondingto a portion of an environment, said first image including anon-occluded image portion corresponding to a portion of the environmentvisible from a first location; generating a first frame including imagecontent from said non-occluded image portion of said first image;receiving an additional image of the environment including at least afirst occluded image portion corresponding to a portion of theenvironment occluded from view from said first location; generating anauxiliary frame including image content from said first occluded imageportion of the additional image; and storing said first frame and saidauxiliary frame in a storage device or transmitting said first frame toanother device.

A method embodiment 17 of operating an image processing system, themethod embodiment 16 further comprising: generating a first UV mapindicating a mapping of portions of said first frame to segments of anenvironmental model which correspond to portions of said environmentwhich are visible from said first location; and generating an auxiliaryUV map indicating a mapping of portions of said auxiliary frame tosegments of an environmental model which correspond to portions of saidenvironment which are not visible from said first location.

A method embodiment 18 of operating an image processing system, themethod embodiment 17 further comprising: generating an environmentalmodel including segments corresponding to non-occluded surfaces of saidenvironment and segments corresponding to occluded surfaces of saidenvironment.

A method embodiment 19 of operating an image processing system, themethod embodiment 16 wherein said auxiliary frame is smaller than saidfirst frame and includes fewer pixels than said first frame.

A method embodiment 20 of operating an image processing system, themethod embodiment 18 further comprising: multiplexing said first frameand said auxiliary frame; and transmitting a multiplexed content streamincluding said first frame in a primary content stream and saidauxiliary frame in an auxiliary content stream.

A method embodiment 21 of operating an image processing system, themethod embodiment 20 wherein multiplexing said first frame and saidauxiliary frame includes incorporating said auxiliary frame in saidmultiplexed content stream before said first frame such that a devicereceiving said multiplexed content stream will receive said auxiliaryframe before said first frame.

A method embodiment 22 of operating an image processing system, themethod embodiment 20, further comprising: receiving a second imagecorresponding to said portion of the environment, said second imageincluding a second non-occluded image portion; receiving a secondadditional image of the environment including at least a second occludedimage portion; and generating a second frame including image contentfrom said second non-occluded image portion of said second image.

A method embodiment 23 of operating an image processing system, themethod embodiment 22 wherein generating the auxiliary frame includes:including image content from said second occluded image portion of thesecond additional image in said auxiliary frame with said first occludedimage portion.

A method embodiment 24 of operating an image processing system, themethod embodiment 23 wherein said auxiliary frame includes occludedimage portions corresponding to M different frames in the primarycontent stream, M being a non-zero integer; and wherein generating theauxiliary frame includes including image content from said secondoccluded image portion of the second additional image in said auxiliaryframe.

A method embodiment 25 of operating an image processing system, themethod embodiment 24 wherein said auxiliary frame is the same size assaid first frame and includes the same number of pixels as said firstframe.

A method embodiment 26 of operating an image processing system, themethod embodiment 16 further comprising: receiving a first request forcontent corresponding to a first program from a first playback devicewhich supports the display of occluded image content; and sending saidfirst frame and the auxiliary frame to the first playback device inresponse to said first request for content.

A method embodiment 27 of operating an image processing system, themethod embodiment 26 further comprising: sending the first UV map andthe auxiliary UV map to the first playback device.

A method embodiment 28 of operating an image processing system, themethod embodiment 26 further comprising: determining from devicecapability information, prior to sending the first frame and theauxiliary frame, that the first playback device supports the display ofoccluded image content.

A method embodiment 29 of operating an image processing system, themethod embodiment 26 further comprising: receiving a second request forcontent corresponding to the first program from a second playback devicewhich does not support the display of occluded image content; andsending said first frame to the second playback device in response tosaid second request for content without sending said auxiliary frame tothe playback second device.

A method embodiment 30 of operating an image processing system, themethod embodiment 26 further comprising: sending the first UV map to thesecond playback device but not sending the auxiliary UV map to theplayback device.

A method embodiment 31 of operating an image processing system, themethod embodiment 29 further comprising: determining from devicecapability information, prior to sending the first frame to the secondplayback device without sending the auxiliary frame that the secondplayback device does not support the display of occluded image content.

An embodiment including a non-transitory computer readable medium foruse in a system, said non-transitory computer readable medium includingcomputer executable instructions which, when executed by a computer,control the system to: receive a first image corresponding to a portionof an environment, said first image including a non-occluded imageportion corresponding to a portion of the environment visible from afirst location; generate a first frame including image content from saidnon-occluded image portion of said first image; receive an additionalimage of the environment including at least a first occluded imageportion corresponding to a portion of the environment occluded from viewfrom said first location; generate an auxiliary frame including imagecontent from said first occluded image portion of the additional image;and store said first frame and said auxiliary frame in a storage deviceor transmit said first frame to another device.

An image processing system embodiment 16 comprising: a receiverconfigured to receive a first image corresponding to a portion of anenvironment, said first image including a non-occluded image portioncorresponding to a portion of the environment visible from a firstlocation; a processor configured to generate a first frame includingimage content from said non-occluded image portion of said first image;wherein said receiver is further configured to receive an additionalimage of the environment including at least a first occluded imageportion corresponding to a portion of the environment occluded from viewfrom said first location; wherein said processor is further configuredto generate an auxiliary frame including image content from said firstoccluded image portion of the additional image; and a memory for storingsaid first frame and said auxiliary frame or a transmitter configured totransmit said first frame to another device.

An image processing system embodiment 17, the image processing systemembodiment 16 wherein said processor is further configured to: generatea first UV map indicating a mapping of portions of said first frame tosegments of an environmental model which correspond to portions of saidenvironment which are visible from said first location; and generate anauxiliary UV map indicating a mapping of portions of said auxiliaryframe to segments of an environmental model which correspond to portionsof said environment which are not visible from said first location.

An image processing system embodiment 18, the image processing systemembodiment 17 wherein said processor is further configured to generatean environmental model including segments corresponding to non-occludedsurfaces of said environment and segments corresponding to occludedsurfaces of said environment.

An image processing system embodiment 19, the image processing systemembodiment 16 wherein said auxiliary frame is smaller than said firstframe and includes fewer pixels than said first frame.

An image processing system embodiment 20, the image processing systemembodiment 18 further comprising: a multiplexer configured to multiplexsaid first frame and said auxiliary frame; and wherein said transmitteris further configured to transmit a multiplexed content stream includingsaid first frame in a primary content stream and said auxiliary frame inan auxiliary content stream.

An image processing system embodiment 21, the image processing systemembodiment 20 wherein said multiplexer is configured, as part of beingconfigured to multiplex said first frame and said auxiliary frame, toincorporate said auxiliary frame in said multiplexed content streambefore said first frame such that a device receiving said multiplexedcontent stream will receive said auxiliary frame before said firstframe.

An image processing system embodiment 22, the image processing systemembodiment 20 wherein said receiver is further configured to receive asecond image corresponding to said portion of the environment, saidsecond image including a second non-occluded image portion; wherein saidreceiver is further configured to receive a second additional image ofthe environment including at least a second occluded image portion; andwherein said processor is further configured to generate a second frameincluding image content from said second non-occluded image portion ofsaid second image.

An image processing system embodiment 23, the image processing systemembodiment 22 wherein said processor is further configured, as part ofbeing configured to generate the auxiliary frame, to include imagecontent from said second occluded image portion of the second additionalimage in said auxiliary frame with said first occluded image portion.

An image processing system embodiment 24, the image processing systemembodiment 23 wherein said auxiliary frame includes occluded imageportions corresponding to M different frames in the primary contentstream, M being a non-zero integer; and wherein said processor isfurther configured, as part of being configured to generate theauxiliary frame, to include image content from said second occludedimage portion of the second additional image in said auxiliary frame.

An image processing system embodiment 25, the image processing systemembodiment 24 wherein said auxiliary frame is the same size as saidfirst frame and includes the same number of pixels as said first frame.

An image processing system embodiment 26, the image processing systemembodiment 16 wherein said receiver is further configured to receive afirst request for content corresponding to a first program from a firstplayback device which supports the display of occluded image content;and wherein said transmitter is further configured to send said firstframe and the auxiliary frame to the first playback device in responseto said first request for content.

An image processing system embodiment 27, the image processing systemembodiment 26 wherein said transmitter is further configured to send thefirst UV map and the auxiliary UV map to the first playback device.

An image processing system embodiment 28, the image processing systemembodiment 26 wherein said processor is further configured to determinefrom device capability information corresponding to the first playbackdevice, prior to the first frame and the auxiliary frame being sent tothe first playback device, that the first playback device supports thedisplay of occluded image content.

An image processing system embodiment 29, the image processing systemembodiment 26 wherein said receiver is further configured to receive asecond request for content corresponding to the first program from asecond playback device which does not support the display of occludedimage content; and wherein said transmitter is further configured tosend said first frame to the second playback device in response to saidsecond request for content without sending said auxiliary frame to theplayback second device.

An image processing system embodiment 30, the image processing systemembodiment 26 wherein said transmitter is further configured to send thefirst UV map to the second playback device but not sending the auxiliaryUV map to the playback device.

An image processing system embodiment 31, the image processing systemembodiment 29 wherein said processor is further configured to determine,from device capability information corresponding to the second playbackdevice, prior to the first frame being sent to the second playbackdevice without the auxiliary frame, that the second playback device doesnot support the display of occluded image content.

A method embodiment 1 of operating a content playback device, the methodcomprising: receiving a first frame including non-occluded image contentcorresponding to a portion of an environment visible from a firstlocation in the environment and occluded image content corresponding toa portion of the environment which is not visible from said firstlocation; detecting a head position of a user; and outputting to adisplay an image of portions of the environment as a function of thedetected head position.

A method embodiment 2 of operating a content playback device, the methodembodiment 1 wherein said outputting to a display an image includes:outputting image portions corresponding to portions of the environmentwhich are visible from the first location when said users head positionis in a default location; and outputting image portions corresponding toat least some portions of the environment which are not visible from thefirst location when said users head position indicates a location offsetfrom said default location.

A method embodiment 3 of operating a content playback device, the methodembodiment 1 further comprising: receiving a UV map indicating a mappingbetween portions of a frame and segments of a model of the environment;and wherein outputting to a display an image includes outputting aportion of a rendered image generated by using the UV map to map imageportions included in the first frame to segments of the model of theenvironment.

A method embodiment 4 of operating a content playback device, the methodembodiment 3 further comprising: receiving said model of theenvironment, said model being a mesh model including segmentscorresponding to surfaces in said environment which are visible from thefirst location and at least some segments corresponding to occludedportions of said environment which are not visible from the firstlocation.

A method embodiment 5 of operating a content playback device, the methodembodiment 3 wherein said non-occluded image content includes contentcaptured by a first camera at said first location in the environment andwherein the occluded image content was captured by an additional cameralocated in said environment at a location offset from said firstlocation.

A method embodiment 6 of operating a content playback device, the methodembodiment 4 further comprising: receiving a list of available programs,said list including for a first program a list of streams correspondingto said program but in different stream formats, a first one of saiddifferent stream formats including a stream format which does notinclude occluded image content and a second stream format which includesoccluded image content; receiving user input selecting said firstprogram; and selecting, based on device capability information, whichstream corresponding to the user selected program to request.

A method embodiment 7 of operating a content playback device, the methodembodiment 6 wherein the playback device supports display of occludedimage content; and wherein said step of selecting includes selecting astream that includes occluded image content; and wherein the methodfurther comprises: sending a request for the selected streamcorresponding to the user selected program to a content server.

A method embodiment 8 of operating a content playback device, the methodembodiment 4 further comprising: receiving a list of available programs,said list including a first program; receiving user input selecting saidfirst program; and sending a request to a content server for the stream.

A method embodiment 9 of operating a content playback device, the methodembodiment 8 further comprising: communicating to said content serverdevice capability information indicating that said playback device iscapable of supporting the display of occluded image contentcorresponding to portions of said environment which are not visible fromsaid first location.

A method embodiment 10 of operating a content playback device, themethod embodiment 9 wherein communicating to said content server devicecapability information includes at least one of: i) providing anidentifier to the content server which can be used to determinecapabilities of the playback device or ii) indicating to the contentserver content stream formats that can be supported by the playbackdevice.

A method embodiment 11 of operating a content playback device, themethod embodiment 10 wherein communicating to said content server devicecapability information includes indicating to the content server thatthe playback device supports a content stream format in which occludedimage data is transmitted in a frame which also includes non-occludedimage data.

An embodiment including a non-transitory computer readable medium foruse in a system, said non-transitory computer readable medium includingcomputer executable instructions which, when executed by a computer,control the system to: receive a first frame including non-occludedimage content corresponding to a portion of an environment visible froma first location in the environment and occluded image contentcorresponding to a portion of the environment which is not visible fromsaid first location; detect a head position of a user; and output to adisplay an image of portions of the environment as a function of thedetected head position.

A content playback device embodiment 1 comprising: a receiver configuredto receive a first frame including non-occluded image contentcorresponding to a portion of an environment visible from a firstlocation in the environment and occluded image content corresponding toa portion of the environment which is not visible from said firstlocation; a processor configured to control said content playback deviceto: detect a head position of a user, and output to a display an imageof portions of the environment as a function of the detected headposition; and a memory for storing said image.

A content playback device embodiment 2, the content playback deviceembodiment 1 wherein output to a display an image includes: outputtingimage portions corresponding to portions of the environment which arevisible from the first location when said users head position is in adefault location; and outputting image portions corresponding to atleast some portions of the environment which are not visible from thefirst location when said users head position indicates a location offsetfrom said default location.

A content playback device embodiment 3, the content playback deviceembodiment 1 wherein said receiver is further configured to receive a UVmap indicating a mapping between portions of a frame and segments of amodel of the environment; and wherein outputting to a display an imageincludes outputting a portion of a rendered image generated by using theUV map to map image portions included in the first frame to segments ofthe model of the environment.

A content playback device embodiment 4, the content playback deviceembodiment 3 wherein said receiver is further configured to receive saidmodel of the environment, said model being a mesh model includingsegments corresponding to surfaces in said environment which are visiblefrom the first location and at least some segments corresponding tooccluded portions of said environment which are not visible from thefirst location.

A content playback device embodiment 5, the content playback deviceembodiment 3 wherein said non-occluded image content includes contentcaptured by a first camera at said first location in the environment andwherein the occluded image content was captured by an additional cameralocated in said environment at a location offset from said firstlocation.

A content playback device embodiment 6, the content playback deviceembodiment 4 wherein said receiver is further configured to receive alist of available programs, said list including for a first program alist of streams corresponding to said program but in different streamformats, a first one of said different stream formats including a streamformat which does not include occluded image content and a second streamformat which includes occluded image content; and said processor isfurther configured to operate said content playback device to: receiveuser input selecting said first program; and select, based on devicecapability information, which stream corresponding to the user selectedprogram to request.

A content playback device embodiment 7, the content playback deviceembodiment 6 further comprising a transmitter configured to send arequest for the selected stream corresponding to the user selectedprogram to a content server, and wherein the playback device supportsdisplay of occluded image content; and wherein said processor as part ofbeing configured to operate said content playback device to select whichstream corresponding to the user selected program to request isconfigured to operate said content playback device to select a streamthat includes occluded image content.

A content playback device embodiment 8, the content playback deviceembodiment 4 further comprising a transmitter configured to send arequest to a content server for the stream; wherein said receiver isfurther configured to receive a list of available programs, said listincluding a first program; and wherein said processor is furtherconfigured to operate said content playback device to receive user inputselecting said first program.

A content playback device embodiment 9, the content playback deviceembodiment 8 wherein said transmitter is further configured tocommunicate to said content server device capability informationindicating that said playback device is capable of supporting thedisplay of occluded image content corresponding to portions of saidenvironment which are not visible from said first location.

A content playback device embodiment 10, the content playback deviceembodiment 9 wherein communicating to said content server devicecapability information includes at least one of: i) providing anidentifier to the content server which can be used to determinecapabilities of the playback device or ii) indicating to the contentserver content stream formats that can be supported by the playbackdevice.

A content playback device embodiment 11, the content playback deviceembodiment 10 wherein communicating to said content server devicecapability information includes indicating to the content server thatthe playback device supports a content stream format in which occludedimage data is transmitted in a frame which also includes non-occludedimage data.

One exemplary embodiment, referred to as a 12th playback methodembodiment, is directed to method of operating a content playbackdevice, in which the method includes the steps of operating the contentplayback device to receive a first frame including non-occluded imagecontent corresponding to a portion of an environment visible from afirst location in the environment, receive an auxiliary frame includingoccluded image content corresponding to a portion of the environmentwhich is not visible from said first location, detecting a head positionof a user; and output to a display image of portions of the environmentas a function of the detected head position. In a thirteenth methodembodiment which includes the features of the 12th method embodiment,the method further includes, as part of outputting to a displayoutputting to the display image portions corresponding to portions ofthe environment which are visible from the first location when said headposition of the user corresponds to a default location and outputting tothe display image portions corresponding to at least some portions ofthe environment which are not visible from the first location when thehead position of the user corresponds to a location offset from saidfirst location.

In a fourteenth playback method embodiment, which includes the featuresof the 12th playback method embodiment the method further comprisesreceiving a first UV map indicating a mapping between portions of thefirst frame including non-occluded image content to segments of a firstmodel of the environment which correspond to portions of saidenvironment visible from said first location; and receiving an auxiliaryUV map indicating a mapping between portions of the auxiliary frameincluding occluded image content to segments of the first model of theenvironment which correspond to portions of said environment which arenot visible from said first location.

In a fifteenth playback method embodiment, which includes the featuresof the fourteenth playback method embodiment, outputting to a displayimage portions of the environment includes, when said user's head ispositioned at a location offset from a default location corresponding tothe first location in the environment: displaying a portion of arendered image generated by i) using the first UV map to map imageportions included in the first frame to segments of the first model ofthe environment and ii) using the auxiliary UV map to map image portionsincluded in the auxiliary frame to segments of the first model of theenvironment which correspond to portions of the environment which arenot visible from said first location. In a sixteenth playback methodembodiment, which includes the features of the fifteenth playback methodembodiment, the method further includes receiving said first frame andauxiliary frame in a multiplexed content stream including a primarycontent stream including said first frame and an auxiliary contentstream including said auxiliary frame.

In a seventeenth playback method embodiment, which includes the featuresof the sixteenth playback method embodiment, the method further includesdemultiplexing said primary content stream and said auxiliary contentstream. In an eighteenth playback embodiment, which includes thefeatures of the seventieth playback embodiment, the primary contentstream includes M frames for each frame in said auxiliary contentstream, M being one or larger. In a nineteenth playback methodembodiment which includes the features of the eighteenth playback methodembodiment, M is greater than one and the auxiliary frame includesoccluded image content corresponding to said first frame and at leastone additional frame in a set of M frames included in said primarycontent stream. In a twentieth playback method embodiment which includesthe features of the nineteenth playback method embodiment, the firstframe and the auxiliary frame are received in encoded form and themethod further includes using a first decoder to decode said first frameprior to image rendering; and using a second decoder which is differentfrom said first decoder to decode the said auxiliary frame prior toimage rendering.

In a twenty-first playback method embodiment, the second decoder isslower than said first decoder. In a twenty second playback methodembodiment, which includes the features of the eighteenth playbackmethod embodiment the auxiliary frame is the same size and includes thesame number of pixels as said first frame. In a twenty third playbackmethod embodiment which includes the features of the eighteenth playbackmethod embodiment, the first frame and the auxiliary frame are receivedin encoded form, and the method further includes using a decoder on atime shared basis to decode said first frame and said auxiliary frameprior to image rendering. In a twenty-fourth playback method embodiment,which includes the features of the fourteenth playback methodembodiment, the method further includes receiving a second environmentalmodel including segments corresponding to portions of the environmentvisible from said first location in the environment, receiving auxiliaryenvironmental model information providing information on segmentscorresponding to occluded portions of said environment which are notvisible from said first location in the environment; and using theauxiliary environmental model information to modify the secondenvironmental model to generate said first environmental model, saidfirst environmental model including segments corresponding to occludedportions of said environment and segments corresponding to non-occludedportions of said environment.

A content playback device embodiment 12 comprising: a receiver forreceiving a first frame including non-occluded image contentcorresponding to a portion of an environment visible from a firstlocation in the environment and for receiving an auxiliary frameincluding occluded image content corresponding to a portion of theenvironment which is not visible from said first location; a memory forstoring received frames; and a processor configured to: detect a headposition of a user; and output, to a display, image of portions of theenvironment as a function of the detected head position.

A content playback device embodiment 13, the content playback deviceembodiment 12 wherein the processor is configured, as part of outputtingto a display: output image portions corresponding to portions of theenvironment which are visible from the first location when said headposition of he user corresponds to a default location; and output imageportions corresponding to at least some portions of the environmentwhich are not visible from the first location when the head position ofthe user corresponds to a location offset from said first location.

A content playback device embodiment 14, the content playback deviceembodiment 12 wherein said receiver is also for: receiving a first UVmap indicating a mapping between portions of the first frame includingnon-occluded image content to segments of a first model of theenvironment which correspond to portions of said environment visiblefrom said first location; and receiving an auxiliary UV map indicating amapping between portions of the auxiliary frame including occluded imagecontent to segments of the first model of the environment whichcorrespond to portions of said environment which are not visible fromsaid first location.

A content playback device embodiment 15, the content playback device ofclaim 14 wherein the processor is configured, as part of outputting to adisplay, when said user's head is positioned at a location offset from adefault location corresponding to the first location in the environment,to: output a portion of a rendered image generated by i) using the firstUV map to map image portions included in the first frame to segments ofthe first model of the environment and ii) using the auxiliary UV map tomap image portions included in the auxiliary frame to segments of thefirst model of the environment which correspond to portions of theenvironment which are not visible from said first location.

A content playback device embodiment 16, the content playback deviceembodiment 15 wherein the receiver receives said first frame andauxiliary frame in a multiplexed content stream including a primarycontent stream including said first frame and an auxiliary contentstream including said auxiliary frame.

A content playback device embodiment 17, the content playback deviceembodiment 16, further comprising: a demultiplexer for demultiplexingsaid primary content stream and said auxiliary content stream.

A content playback device embodiment 18, the content playback deviceembodiment 17 wherein said primary content stream includes M frames foreach frame in said auxiliary content stream, M being one or larger.

A content playback device embodiment 19, the content playback deviceembodiment 18 wherein M is greater than one, said auxiliary frameincludes occluded image content corresponding to said first frame and atleast one additional frame in a set of M frames included in said primarycontent stream.

A content playback device embodiment 20, the content playback deviceembodiment 19 wherein said first frame and said auxiliary frame arereceived in encoded form, and wherein said content playback deviceincludes: a first decoder to decode said first frame prior to imagerendering; and a second decoder which is different from said firstdecoder to decode the said auxiliary frame prior to image rendering.

A content playback device embodiment 21, the content playback deviceembodiment 20 wherein said second decoder is slower than said firstdecoder.

A content playback device embodiment 22, the content playback deviceembodiment 18 wherein said first frame and said auxiliary frame arereceived in encoded form, the method further comprising: using a decoderon a time shared basis to decode said first frame and said auxiliaryframe prior to image rendering.

A content playback device embodiment 23, the content playback deviceembodiment 14 wherein said receiver is also for: receiving a secondenvironmental model including segments corresponding to portions of theenvironment visible from said first location in the environment;receiving auxiliary environmental model information providinginformation on segments corresponding to occluded portions of saidenvironment which are not visible from said first location in theenvironment; and using the auxiliary environmental model information tomodify the second environmental model to generate said firstenvironmental model, said first environmental model including segmentscorresponding to occluded portions of said environment and segmentscorresponding to non-occluded portions of said environment.

Another exemplary embodiment includes a non-transitory computer readablemedium having computer executable instructions stored thereon which,when executed by a processor of a content playback device control thecontent playback device to perform the steps of: receiving a first frameincluding non-occluded image content corresponding to a portion of anenvironment visible from a first location in the environment; receivingan auxiliary frame including occluded image content corresponding to aportion of the environment which is not visible from said firstlocation; detecting a head position of a user; and outputting to adisplay image of portions of the environment as a function of thedetected head position.

While steps are shown in an exemplary order it should be appreciatedthat in many cases the order of the steps may be altered withoutadversely affecting operation. Accordingly, unless the exemplary orderof steps is required for proper operation, the order of steps is to beconsidered exemplary and not limiting.

While various embodiments have been discussed, it should be appreciatedthat not necessarily all embodiments include the same features and someof the described features are not necessary but can be desirable in someembodiments.

While various ranges and exemplary values are described the ranges andvalues are exemplary. In some embodiments the ranges of values are 20%larger than the ranges discussed above. In other embodiments the rangesare 20% smaller than the exemplary ranges discussed above. Similarly,particular values may be, and sometimes are, up to 20% larger than thevalues specified above while in other embodiments the values are up to20% smaller than the values specified above. In still other embodimentsother values are used.

The techniques of various embodiments may be implemented using software,hardware and/or a combination of software and hardware. Variousembodiments are directed to apparatus, e.g., a image data capture andprocessing systems. Various embodiments are also directed to methods,e.g., a method of image capture and/or processing image data. Variousembodiments are also directed to a non-transitory machine, e.g.,computer, readable medium, e.g., ROM, RAM, CDs, hard discs, etc., whichinclude machine readable instructions for controlling a machine toimplement one or more steps of a method.

Various features of the present invention are implemented using modules.Such modules may, and in some embodiments are, implemented as softwaremodules. In other embodiments the modules are implemented in hardware.In still other embodiments the modules are implemented using acombination of software and hardware. In some embodiments the modulesare implemented as individual circuits with each module beingimplemented as a circuit for performing the function to which the modulecorresponds. A wide variety of embodiments are contemplated includingsome embodiments where different modules are implemented differently,e.g., some in hardware, some in software, and some using a combinationof hardware and software. It should also be noted that routines and/orsubroutines, or some of the steps performed by such routines, may beimplemented in dedicated hardware as opposed to software executed on ageneral purpose processor. Such embodiments remain within the scope ofthe present invention. Many of the above described methods or methodsteps can be implemented using machine executable instructions, such assoftware, included in a machine readable medium such as a memory device,e.g., RAM, floppy disk, etc. to control a machine, e.g., general purposecomputer with or without additional hardware, to implement all orportions of the above described methods. Accordingly, among otherthings, the present invention is directed to a machine-readable mediumincluding machine executable instructions for causing a machine, e.g.,processor and associated hardware, to perform one or more of the stepsof the above-described method(s).

Some embodiments are directed a non-transitory computer readable mediumembodying a set of software instructions, e.g., computer executableinstructions, for controlling a computer or other device to encode andcompresses stereoscopic video. Other embodiments are embodiments aredirected a computer readable medium embodying a set of softwareinstructions, e.g., computer executable instructions, for controlling acomputer or other device to decode and decompresses video on the playerend. While encoding and compression are mentioned as possible separateoperations, it should be appreciated that encoding may be used toperform compression and thus encoding may, in some include compression.Similarly, decoding may involve decompression.

In various embodiments a processor of a processing system is configuredto control the processing system to perform the method steps performedby the exemplary described processing system. In various embodiments aprocessor of a playback device is configured to control the playbackdevice to implement the steps, performed by a playback device, of one ormore of the methods described in the present application.

Numerous additional variations on the methods and apparatus of thevarious embodiments described above will be apparent to those skilled inthe art in view of the above description. Such variations are to beconsidered within the scope.

What is claimed:
 1. A method of operating a content playback device, themethod comprising: receiving a first frame including non-occluded imagedata corresponding to a first portion of an environment visible from afirst viewing position; receiving an auxiliary frame including occludedimage data corresponding to a second portion of the environment notvisible from the first viewing position and visible from a secondviewing position; detecting a head position; rendering an image based onthe detected head position, the non-occluded image data, and theoccluded image data, wherein the rendered image comprises a portion ofthe environment from the detected head position; and outputting therendered image to a display.
 2. The method of claim 1, furthercomprising: outputting, to the display, image portions corresponding toportions of the environment which are visible from the head positionwhen the head position is in a default location.
 3. The method of claim1,wherein rendering the image further comprises: receiving a first UVmap indicating a mapping between portions of the first frame includingnon-occluded image content to segments of a first model of theenvironment which correspond to portions of said environment visiblefrom the detected head position; and receiving an auxiliary UV mapindicating a mapping between portions of the auxiliary frame includingoccluded image content to segments of the first model of the environmentwhich correspond to portions of said environment which are not visiblefrom the detected head position.
 4. The method of claim 3, furthercomprising: receiving said first frame and auxiliary frame in amultiplexed content stream including a primary content stream includingsaid first frame and an auxiliary content stream including saidauxiliary frame.
 5. The method of claim 4, further comprising:demultiplexing said primary content stream and said auxiliary contentstream.
 6. The method of claim 5, wherein said first frame and saidauxiliary frame are received in encoded form, the method furthercomprising: using a first decoder to decode said first frame prior toimage rendering; and using a second decoder which is different from saidfirst decoder to decode the said auxiliary frame prior to imagerendering.
 7. The method of claim 6, wherein said auxiliary frame is asame size and includes a same number of pixels as said first frame.
 8. Anon-transitory computer readable medium comprising computer readablecode executable by one or more processors to: receive a first frameincluding non-occluded image data corresponding to a first portion of anenvironment visible from a first viewing position; receive an auxiliaryframe including occluded image data corresponding to a second portion ofthe environment not visible from the first viewing position and visiblefrom a second viewing position; detect a head position; render an imagebased on the detected head position, the non-occluded image data, andthe occluded image data, wherein the rendered image comprises a portionof the environment from the detected head position; and output therendered image to a display.
 9. The non-transitory computer readablemedium of claim 8, further comprising computer readable code to: output,to the display, image portions corresponding to portions of theenvironment which are visible from the head position when the headposition is in a default location.
 10. The non-transitory computerreadable medium of claim 8, wherein the computer readable code to renderthe image further comprises computer readable code to: receive a firstUV map indicating a mapping between portions of the first frameincluding non-occluded image content to segments of a first model of theenvironment which correspond to portions of said environment visiblefrom the detected head position; and receive an auxiliary UV mapindicating a mapping between portions of the auxiliary frame includingoccluded image content to segments of the first model of the environmentwhich correspond to portions of said environment which are not visiblefrom the detected head position.
 11. The non-transitory computerreadable medium of claim 10, further comprising computer readable codeto: receive said first frame and auxiliary frame in a multiplexedcontent stream including a primary content stream including said firstframe and an auxiliary content stream including said auxiliary frame.12. The non-transitory computer readable medium of claim 11, furthercomprising computer readable code to: demultiplex said primary contentstream and said auxiliary content stream.
 13. The non-transitorycomputer readable medium of claim 12, wherein said first frame and saidauxiliary frame are received in encoded form, the non-transitorycomputer readable medium further comprising computer readable code to:use a first decoder to decode said first frame prior to image rendering;and use a second decoder which is different from said first decoder todecode the said auxiliary frame prior to image rendering.
 14. Thenon-transitory computer readable medium of claim 13, wherein saidauxiliary frame is a same size and includes a same number of pixels assaid first frame.
 15. A system comprising: one or more processors; andone or more non-transitory computer readable media comprising computerreadable code executable by the one or more processors to: receive afirst frame including non-occluded image data corresponding to a firstportion of an environment visible from a first viewing position; receivean auxiliary frame including occluded image data corresponding to asecond portion of the environment not visible from the first viewingposition and visible from a second viewing position; detect a headposition; render an image based on the detected head position, thenon-occluded image data, and the occluded image data, wherein therendered image comprises a portion of the environment from the detectedhead position; and output the rendered image to a display.
 16. Thesystem of claim 15, further comprising computer readable code to:output, to the display, image portions corresponding to portions of theenvironment which are visible from the head position when the headposition is in a default location.
 17. The system of claim 15, whereinthe computer readable code to render the image further comprisescomputer readable code to: receive a first UV map indicating a mappingbetween portions of the first frame including non-occluded image contentto segments of a first model of the environment which correspond toportions of said environment visible from the detected head position;and receive an auxiliary UV map indicating a mapping between portions ofthe auxiliary frame including occluded image content to segments of thefirst model of the environment which correspond to portions of saidenvironment which are not visible from the detected head position. 18.The system of claim 17, further comprising computer readable code to:receive said first frame and auxiliary frame in a multiplexed contentstream including a primary content stream including said first frame andan auxiliary content stream including said auxiliary frame.
 19. Thesystem of claim 18, further comprising computer readable code to:demultiplex said primary content stream and said auxiliary contentstream.
 20. The system of claim 19, wherein said first frame and saidauxiliary frame are received in encoded form, the non-transitorycomputer readable media further comprising computer readable code to:use a first decoder to decode said first frame prior to image rendering;and use a second decoder which is different from said first decoder todecode the said auxiliary frame prior to image rendering.